Prompting Large Language Models with Audio for General-Purpose Speech Summarization

Wonjune Kang, Deb Roy

Abstract

In this work, we introduce a framework for speech summarization that leverages the processing and reasoning capabilities of large language models (LLMs). We propose an end-to-end system that combines an instruction-tuned LLM with an audio encoder that converts speech into token representations that the LLM can interpret. Using a dataset with paired speech-text data, the overall system is trained to generate consistent responses to prompts with the same semantic information regardless of the input modality. The resulting framework allows the LLM to process speech inputs in the same way as text, enabling speech summarization by simply prompting the LLM. Unlike prior approaches, our method is able to summarize spoken content from any arbitrary domain, and it can produce summaries in different styles by varying the LLM prompting strategy. Experiments demonstrate that our approach outperforms a cascade baseline of speech recognition followed by LLM text processing.

Contents

This page contains examples of spoken document summaries generated using the method proposed in the above paper. We provide sample summaries of articles from the CNN / DailyMail Dataset, where speech has been synthesized using a StyleTTS 2 model trained on the LJ Speech dataset, as well as examples demonstrating how varying the prompt to the LLM can produce summaries in different styles.

We also demonstrate summarization of real audio that was recorded in “in-the-wild” settings (stories spoken by actual people), demonstrating the generalizability of our method to real-life scenarios.

Finally, we show examples of our system’s responses when using spoken questions from the human-spoken HeySQuAD dataset as prompts, demonstrating its ability to utilize its capabilities as an LLM while understanding and processing general speech inputs.

All outputs shown below from our proposed system were generated using speech inputs.

Summarization Examples: CNN / DailyMail

Article 1

Audio (StyleTTS 2)

Text

"Juventus moved ever closer to clinching a fourth straight Serie A title by beating nearest challengers Lazio 2-0 in Turin. First-half goals from Carlos Tevez - his 26th of the season in all competitions - and centre-back Leonardo Bonucci pushed Juve 15 points clear of second place with only seven games remaining. Lazio did mount a late rally at the Juventus Stadium but the dismissal of midfielder Danilo Cataldi for a heavy tackle on Tevez ended their challenge. Carlos Tevez opened the scoring for Juventus after 17 minutes as they beat Lazio on Saturday . Tevez jumps for joy after firing Juventus ahead with his 26th goal of the season . The result brought Lazio's run of six consecutive victories to an end and the Biancocelesti could be dragged back into third place should their city rivals Roma beat Atalanta in the capital on Sunday. Earlier on Saturday, ambitious Sampdoria were held to an unlikely goalless draw by Cesena but used the point to take a share of fourth place. The Genoa club dominated their relegation-threatened opponents from start to finish but found goalkeeper Federico Agliardi an immovable object when their radar was not malfunctioning. Samp are now level on Napoli with 60 points while 18th-placed Cesena are six points shy of Atalanta and safety with seven games still to play."

Reference summary from CNN / DailyMail

"Juventus beat Lazio 2-0 on Saturday to move 15 points clear in Serie A . Carlos Tevez and Leonardo Bonucci scored in the first half for Old Lady . Earlier in the day Sampdoria were held to a draw by lowly Cesena ."

Reference summary generated by Llama 2-Chat 7B

"Juventus moved 15 points clear of second place in Serie A with a 2-0 win over Lazio, thanks to goals from Carlos Tevez and Leonardo Bonucci. Tevez's 26th goal of the season gave Juve an early lead, and a late tackle by Lazio's Danilo Cataldi resulted in a red card and ended their comeback hopes. The result moves Juventus closer to their fourth consecutive Serie A title, while Lazio's loss ends their six-match winning streak."

Summaries generated by proposed system

Prompt 1: "Summarize the following article in 3 sentences or less: "

"Juventus is close to winning a fourth straight Serie A title by defeating Lazio 2-0. Carlos Tevez scored his 26th goal of the season, helping Juventus take an early lead. Lazio attempted a late comeback but were unable to score, ending their six-game winning streak."

Prompt 2: "Summarize the following article in 3 sentences or less without mentioning any names: "

"Juventus's 2-0 victory over Lazio moved them closer to their fourth consecutive Serie A title. Lazio's 15-point lead over second place was reduced to 19 points after the match. Sampdoria drew 0-0 against Cesena, leaving both teams level on 60 points."

Prompt 3: "Summarize the following article in 3 sentences or less, focusing on Lazio: "

"Carlos Tevez scored twice as Juventus defeated Lazio 2-0, moving 15 points ahead of second-place rivals. Lazio's six-game winning streak ended, and the team could be dragged back into third place if their city rivals, Roma, beat Atalanta. Sampdoria drew 0-0 against Cesena, leaving both teams level on 60 points."

Article 2

Audio (StyleTTS 2)

Text

"(CNN)The much-discussed trailer for "Batman v. Superman: Dawn of Justice" just got a makeover. Soon after the trailer went up late on Friday, YouTuber Bobby Burns got to work on doing his own version using classic superhero footage. Instead of Ben Affleck and Henry Cavill facing off, this version wonders what it would be like if comics' classic characters Adam West and Christopher Reeve took their places. CNN spoke to 18-year-old Burns about what motivated him to make his now-viral video. (Warner Bros. Pictures, the studio behind "Batman v. Superman," is owned by Time Warner, as is CNN.) CNN: When did you start making YouTube videos like this? Burns: At age 11. I work for the Made In Network in Nashville, Tennessee. CNN: Where did this idea come from? Burns: I'm pretty hyped for "Batman v. Superman." As soon as I saw the trailer, my brain immediately went to the classic "Batman" [TV series] and I thought how funny it would be if this film was made at the time. CNN: What were you looking to accomplish? Burns: I wanted to contrast the old with the new. CNN: What kind of reaction have you received? Burns: Within an hour of the trailer going up, I went to work on the edit. Within four hours, my edit was finished and up on the Internet. The reaction has been great! It's awesome to see so many people enjoying what I made. Take a look at the fan trailer below: ."

Reference summary from CNN / DailyMail

"A fan re-edited "Batman v. Superman: Dawn of Justice" trailer with classic scenes from older "Batman" and "Superman" TV and film . Adam West and Christopher Reeve replace Ben Affleck and Henry Cavill in the re-imagined trailer ."

Reference summary generated by Llama 2-Chat 7B

"A YouTuber named Bobby Burns created a viral fan trailer for "Batman v. Superman: Dawn of Justice" using classic superhero footage from the 1960s "Batman" TV series and the 1978 "Superman" movie starring Christopher Reeve. Burns, 18, was motivated by his enthusiasm for the film and a desire to contrast the old with the new. The reaction to the fan trailer has been positive, with many viewers enjoying the creative take on the movie's trailer."

Summaries generated by proposed system

Prompt 1: "Summarize the following article in 3 sentences or less: "

"Bobby Burns, a 18-year-old from Nashville, Tennessee, was inspired to create his Batman v Superman Dawn of Justice trailer makeover after watching the original trailer. He used classic superhero footage instead of Ben Affleck and Henry Cavill and imagined characters like Adam West and Christopher Reeve in the roles. The reaction to his work has been positive, with many people enjoying the contrast between the old and new Batman."

Prompt 2: "Summarize the following article, focusing on the content in the quotes: "

"Bobby Burns, a 18-year-old from Nashville, Tennessee, was inspired to create his Batman v Superman Dawn of Justice trailer makeover after watching the original Batman TV series. He wanted to contrast the old with the new and created a version using classic superhero footage instead of Ben Affleck and Henry Cavill. The reaction to his work has been great, with many people enjoying it. Burns worked on the edit within four hours of the original trailer going up and it was up on the internet within an hour. The reaction has been positive, with many people enjoying his version."

Article 3

Audio (StyleTTS 2)

Text

"(CNN)Israeli Prime Minister Benjamin Netanyahu will get two extra weeks to form a government, Israel's President said in a news release Monday. Netanyahu made the request at President Reuven Rivlin's Jerusalem home Monday. "We have made progress and are on the way to forming a government," Netanyahu told Rivlin, according to a statement. "However, I require additional time in order for the government to be stable and so that we might reach agreement on important issues that will aid us in meeting the challenges facing the State of Israel." Netanyahu must form his government in less than 42 days, according to Israeli law. "I wish you success in your work," Rivlin told the Prime Minister, according to a statement. "The entire people of Israel hope that a government will be established; indeed, a transition government has not received the confidence of the Knesset and is viewed by the public as needing to be dealt with. I hope that in the coming days you will succeed in forming a stable government for the State of Israel." Last month, Netanyahu's Likud Party snared 30 of the 120 seats in the Knesset, Israel's parliament, giving its leaders the first chance at forming a coalition government. The Zionist Union came in second, with 24 seats."

Reference summary from CNN / DailyMail

"Israeli law says the Prime Minister must form his government in less than 42 days . Netanyahu cites government stability and reaching "agreement on important issues" as reasons he needs additional time ."

Reference summary generated by Llama 2-Chat 7B

"Israeli Prime Minister Benjamin Netanyahu has been granted an additional two weeks to form a government after meeting with President Reuven Rivlin on Monday. Netanyahu requested the extra time to ensure the government is stable and to address important issues facing the country. Rivlin wished Netanyahu success in his efforts, expressing the hope that a stable government will be established."

Summaries generated by proposed system

Prompt 1: "Summarize the following article in 3 sentences or less: "

"Israeli Prime Minister Benjamin Netanyahu has been granted an additional two weeks to form a government. President Reuven Rivlin expressed hope that a stable government would be established soon, as the current transitional government has not received the confidence of the Nesset and is viewed by the public as needing to be dealt with. Netanyahu's Likud party won 30 of the 120 seats in the Nesset, giving it the first chance at forming a coalition government, while the Zionist Union came in second with 24 seats."

Prompt 2: "Summarize the following article in 8 sentences or less: "

"Israeli Prime Minister Benjamin Netanyahu has been granted an additional two weeks to form a government. President Reuven Rivlin made the request at Netanyahu's Jerusalem home, stating that progress has been made but more time is needed to form a stable government. Netanyahu requires additional time to address important issues that will aid in meeting Israel's challenges. He must form his government within 42 days, as per Israeli law. Rivlin expressed hope that a stable government will be established, as the current transitional government has not received public confidence. Netanyahu's Likud party won 30 of the 120 seats in the Knesset, giving them the first opportunity to form a coalition government. The Zionist Union came in second with 24 seats."

Prompt 3: "Summarize the following article, focusing on the content in the quotes: "

"Israeli Prime Minister Benjamin Netanyahu has been granted an additional two weeks to form a government, according to President Reuven Rivlin. Netanyahu made the request at a meeting at Rivlin's Jerusalem home on Monday. The president stated that progress has been made and the government is on the way to being formed. However, Netanyahu requested additional time to ensure a stable government and to reach agreements on important issues that will aid in meeting the challenges facing Israel. Netanyahu must form his government within 42 days, according to Israeli law. Rivlin told Netanyahu that the entire people of Israel hope that a government will be established."

Back to top

Summarization Examples: Real Speech Data

The following shows examples of summaries produced by our system when feeding in real spoken utterances recorded in “in-the-wild” settings. In both examples, the speakers are sharing a personal life experience.

Sample 1 (Female Speaker)

Audio

Transcript

"I feel like I'm living on borrowed time. I have a limited amount of savings, and probably around January or February, I won't have enough money to continue living in my apartment. And my apartment is one of the cheapest in Fayetteville. I got it six years ago, and the rent hasn't gone up too much. I rent from a private landlady. She has been pretty good through this whole pandemic. However, I'm still really worried about my future. For a really long time I was looking into buying RVs, trailers, shacks, living out of my minivan. I'm really concerned and I'm worried that nobody really understands the effect of this, like the way this is going to impact millions of people. We need more affordable housing. We need more laws that protect renters. We need to have more support for landlords. When they are struggling with tenants who can't make their rent, it's like a trickle down effect. And I think it's really important to look at this from a perspective of how can we lift everybody up rather than punish people who have been impacted by circumstances entirely out of their control. I know for me personally, thinking about this is really difficult emotionally because I don't want to lose my home, but I probably will. It's just a matter of time."

Summaries generated by proposed system

Prompt 1: "Summarize the following story: "

"The speaker feels like they are living on borrowed time due to limited savings and the possibility of not being able to afford their apartment rent in January or February. They are concerned about the future and the impact of the pandemic on affordable housing, renters, and landlords. The speaker believes that more support and laws should be in place to protect renters and landlords struggling with tenants who cannot make their rent. They feel that it is important to look at the issue from a perspective of how to lift everyone up rather than punishing people who have been impacted by circumstances out of their control. The speaker is emotionally challenged by the situation, as they do not want to lose their home but are aware that it may be inevitable in the future."

Prompt 2: "Summarize the following story in the first person: "

"I feel like I'm living on borrowed time. I have limited savings and probably won't have enough money to continue living in my apartment by January or February. My apartment is one of the cheapest in Fayetteville, and I rent from a private landlady who has been pretty good during this pandemic. However, I'm still really worried about my future. I was looking into buying a tiny home, but I'm really concerned and worried that nobody really understands the impact of this. We need more affordable housing, more laws that protect renters, and more support for landlords who are struggling with tenants who can't make their rent. It's a trickle-down effect, and I think it's really important to look at this from a perspective of how can we lift everybody up rather than punish people who have been impacted by circumstances entirely out of their control. I know thinking about this is really difficult emotionally because I don't want to lose my home, but it's just a matter of time."

Sample 2 (Male Speaker)

Audio

Transcript

"My personal experiences that was with, when I was in high school everybody wanted me to play basketball like my brother, who was a basketball star, but nobody cared what sport I really wanted to play and where my heart was and what my love was, and I relate that to, now, as I coach baseball, I see a lot of these parents, a lot of these fathers trying to live vicariously through their sons, and daughters, that they bring to play baseball, and they could care less what their child actually wants to play, and so the impact of that is these kids not putting their actual love into the sport. And, you know, it creates tension in the home because these parents are trying to live vicariously through them. And they yelling at them, being that parent that's yelling from the stands, wanting them to be a certain way. And these kids, their heart is not in it. So, I think we need to care a lot more about the mental health of these young adolescents, these young pre-teens, and see what's really on their mind. And that's my personal experience."

Summaries generated by proposed system

Prompt 1: "Summarize the following story: "

"The speaker shares their personal experience of being a young athlete, comparing it to their current role as a coach. They mention how many parents push their children to play sports, especially baseball, despite the children's true interests. The speaker believes that this pressure leads to tension at home, as parents try to control their children's lives through sports and yelling at them. They also mention that the children's hearts are not in the sport, and they think that we need to care more about the mental health of these young athletes, especially those who are less experienced."

Prompt 2: "Summarize the following story in the first person: "

"I have personal experience with parents who try to force their children to play basketball, despite their children's true passions. I relate this to the current situation in baseball, where many parents push their children to play baseball, even if their children don't have a love for the sport. This creates tension at home, as parents try to live by their expectations and yell at their children if they don't perform well. I believe we need to care more about the mental health of these young athletes and provide them with the necessary support and guidance to help them find their passion. It's important to remember that this is a common experience, and it's essential to understand and empathize with the struggles that young athletes face."

Back to top

Responses to General Speech Prompts: Spoken Question Answering on HeySQuAD

Example 1

Audio

Question Transcript

"Combustion is caused by an oxidant and a fuel. What role does oxygen play in combustion?"

LLM Response

"Oxygen plays a crucial role in combustion, which is the process of burning a fuel in the presence of oxygen to produce heat, light, and various byproducts.

In combustion, oxygen serves as the oxidizing agent, meaning it provides the energy needed to break down the fuel molecules into their constituent elements (such as carbon dioxide, water vapor, and other gases). The chemical reaction can be represented as:

fuel + O2 → products + heat

Here, fuel can be any organic or inorganic substance that can be burned, such as gasoline, wood, coal, or methane. Oxygen is essential for the combustion process to occur, as it reacts with the fuel to produce heat and light, which can be harnessed for various purposes, such as powering engines, generating electricity, or cooking food.

In summary, oxygen plays a vital role in combustion by providing the energy required to break down the fuel molecules into their constituent elements, resulting in the release of heat, light, and various byproducts."

Example 2

Audio

Question Transcript

"What is the highest court in European Union Law?"

LLM Response

"The highest court in the European Union (EU) is the European Court of Justice (ECJ). The ECJ is the primary judicial institution of the EU and is responsible for interpreting EU law and ensuring that EU member states comply with EU law. The ECJ is composed of 27 judges who are appointed by the national courts of each EU member state. The ECJ is based in Luxembourg and operates as a court of last resort, meaning that its decisions are binding on all EU member states. The ECJ's decisions are final and non-appealable, and they are considered to be the highest authority in EU law.

It's worth noting that the European Court of Human Rights (ECHR) is another important institution in the EU legal system, it is the highest court for human rights in the EU and it is responsible for interpreting the European Convention on Human Rights and the European Social Charter. The ECHR is based in Strasbourg, France and it is an independent judicial institution, but it is not considered as the highest court in the EU legal system.

In summary, the European Court of Justice (ECJ) is the highest court in the European Union (EU) and it is responsible for interpreting EU law and ensuring that EU member states comply with EU law. The European Court of Human Rights (ECHR) is another important institution in the EU legal system, it is the highest court for human rights in the EU and it is responsible for interpreting the European Convention on Human Rights and the European Social Charter."

Example 3

Audio

Question Transcript

"Why does competition among workers drive down wages?"

LLM Response

"The relationship between competition among workers and wages is complex and can be influenced by various factors. One of the primary reasons competition drives down wages is the concept of supply and demand in the labor market. When there is high competition for jobs, employers have more options and can choose the workers they want to hire at the lowest possible cost. This can lead to employers offering lower wages to attract workers and maintain a competitive workforce.

Additionally, competition can lead to a race to the bottom, where employers try to keep wages low to attract workers and stay competitive. This can result in employers offering lower wages, fewer benefits, and less job security, which can ultimately lead to lower overall wages for all workers.

Another factor is the role of unions. When unions are strong and workers have the power to negotiate better wages and working conditions, employers may be less likely to engage in wage competition. However, when unions are weak or non-existent, workers may not have the same bargaining power, and employers may be more likely to engage in wage competition to attract workers.

Furthermore, technological advancements and automation can also contribute to wage competition. As machines and software become more advanced, they can perform tasks that were previously done by humans, leading to job displacement and a need for workers to compete for fewer jobs. This can result in employers offering lower wages to attract workers and maintain a competitive workforce.

It's important to note that not all cases of competition among workers result in wage downsizing. In some cases, competition can lead to innovation, increased productivity, and higher wages for workers. However, the general trend of competition driving down wages is a significant concern in the labor market today."

Back to top