Turning Audio Data into Actionable Insights with Azure

In today’s data-driven landscape, organizations collect large amounts of audio data that contain valuable insights. However, its unstructured format makes it challenging to analyze effectively, limiting the ability to leverage this information for business improvement. Fortunately, advances in digital tools have made it easier to unlock the value of voice data. Speech-to-Text (STT) technology plays a key role by converting spoken words into searchable, analyzable text, opening up new possibilities for business intelligence, operational efficiency, and enhanced customer experience.

In this post, we’ll explore key use cases enabled by STT and their business value. Then, we’ll dive into a technical guide on how to build a STT-powered pipeline in Azure, with a focus on how it can enable efficient knowledge sharing at ML6.

1. Key use cases enabled by STT and their business value (2-minute read).
2. A technical deep dive into the essential components for building a STT-powered pipeline in Azure (5-minute read).

The business value of modern STT solutions

Challenges with audio data analysis

Companies generate large amounts of audio data from sources like customer interactions, meetings, support calls, sales pitches, interviews, and more. This presents a significant opportunity to extract valuable insights from raw audio. However, many organizations struggle to harness the full potential of this data. According to this article, businesses traditionally faced three key challenges in understanding the voice of the customer through audio data:

1. Incomplete sampling: Companies often use manual random sampling from their broad data pool, capturing less than 2% of interactions. This leads to incomplete and unrepresentative datasets, which weakens the accuracy of subsequent analysis.
2. Inaccurate transcriptions: Outdated STT systems often struggle with accents, speech patterns, and background noise, limiting the usefulness of the extracted information.
3. Lack of actionable insights: Even with accurate transcriptions, organizations often fail to derive actionable insights, missing opportunities to identify trends and drive meaningful outcomes.

Advancements in audio analysis

So what has changed? The landscape of audio data analysis has been transformed by breakthroughs in STT and language models, providing remarkable accuracy and uncovering deeper insights:

• State-of-the-art deep learning architectures, such as OpenAI’s Whisper, have revolutionized STT by surpassing the limitations of older techniques such as Hidden Markov Model (HMM)-based approaches, enabling highly accurate transcription across various languages and accents.
• Large Language Models (LLMs) address both the issue of incomplete sampling and the challenge of deriving actionable insights. By leveraging their advanced language understanding, contextual awareness, and capacity to handle enormous amounts of textual data, LLMs can work with larger datasets, reducing the impact of incomplete sampling and transform raw transcriptions into concise summaries, trend analyses, and other valuable insights, thereby making audio data meaningful for business outcomes.

These advancements empower organizations to unlock deeper insights from large amounts of audio data, enabling them to access valuable information that they previously struggled to obtain. The figure below shows various use cases that accurate STT technology can unlock in combination with LLMs.

STT use cases

Use case: Enhancing knowledge sharing

At ML6, knowledge sharing is at the heart of what we do. For example, we have our Insights Hub, a central platform where we document and store key learnings from our projects. On top of that, we have Skip, our smart chatbot, which is seamlessly integrated with resources like the Insights Hub, allowing us to instantly access answers to specific questions and past experiences.

To foster continuous learning and improvement, we regularly hold learning sessions in various formats, such as workshops and presentations, where our team members come together to present and discuss important lessons learned. These sessions cover a wide range of topics, from technical insights gained from projects utilizing cutting-edge technology to functional knowledge related to areas like business development. To ensure that these valuable insights are captured and disseminated throughout the team, all sessions are recorded. Building on this, we could further enhance our capabilities through automated transcription:

• Transcribe knowledge-sharing presentations using STT: Recorded presentations can be transcribed using STT technology (e.g., Whisper), turning unstructured audio data into a searchable and accessible format.
• Extract key insights and summaries: The transcriptions can then be processed to extract key insights and generate concise summaries, making it easy to reference and share learnings across the team.

By automating the transcription and summarization of our knowledge-sharing content, we can ensure that valuable information is preserved, easily accessible, and ready to drive future innovation.

Implementing a STT-powered pipeline in Azure

In this section, we provide a high-level guide to implementing a pipeline in Azure that automatically processes incoming audio recordings from knowledge-sharing presentations by transcribing and processing them, specifically for creating summaries and extracting key insights. A high-level overview can be seen in the figure below.

High level Azure pipeline

AI model comparison

At the heart of the pipeline lies the STT model used to transcribe audio recordings. Azure AI services offers a broad range of STT-capabilities. These options can be accessed through Azure Speech Service, featuring STT models developed by Microsoft, or through Azure’s OpenAI Services, which provide access to OpenAI models like Whisper — the current leader in STT technology in terms of accuracy. While we won’t delve into the technical details of Whisper here, those curious about its inner workings can explore more in this OpenAI post.

Azure OpenAI’s Whisper model stands out for its exceptional accuracy and cost-effectiveness, making it a strong choice for many applications. However, Azure Speech Service provides additional features such as speaker diarization and word-level timestamps, which are essential for more complex transcription needs. Interestingly, Whisper models are also available via Azure Speech, allowing users to combine Whisper’s accuracy with these advanced features.

The flowchart below helps you choose the right service based on your specific use case. Of course, additional factors like request quotas and limits should also be considered, for which refer to the following documentation. For our use case, we opted for Azure’s OpenAI Whisper due to its high accuracy, low cost, and suitability for building an event-triggered pipeline.

Azure STT flowchart

As an open-source model, Whisper can also be self-hosted, providing benefits like greater control over deployment, improved data privacy, and the ability to fine-tune the model with domain-specific data. However, self-hosting also has its downsides, including the high initial costs associated with setting up the necessary infrastructure. While this topic isn’t central to the current pipeline, those interested in deployment strategies and fine-tuning options for self-hosting Whisper can refer to articles by some of my colleagues for additional insights (1, 2).

For extracting insights and summaries from the shared learnings, we use an LLM, specifically GPT-4o. The latter is also available through Azure’s OpenAI service. With well-crafted prompt engineering, you can extract valuable insights, without the additional effort and resources required to finetune a model. However, the effectiveness depends on the specific use case, and you may need to go through some trial and error to improve the results accordingly. The following article provides valuable insights to help you decide whether fine-tuning is the right approach for your specific use case.

Azure technology stack overview

The Azure solution, of which a figure is shown further down the page, leverages several key Azure services. Each technology is selected for its ability to effectively contribute to the overall pipeline, ensuring that the workflow remains automated, efficient, and easy to maintain:

• Azure Blob Storage: Used to store incoming audio recordings as well as the insights and summaries from the presentation. Blob Storage is ideal for storing large amounts of unstructured data, such as multimedia files, and acts as the repository for both the input and output of the pipeline.
• Azure Event Grid: Facilitates real-time processing by acting as an event trigger when new audio files are uploaded to Blob Storage. This ensures immediate initiation of the transcription process, automating the workflow without manual intervention.
• Azure Functions: Provides serverless compute to handle the transcription and summarization workflows. Azure Functions are chosen for their scalability, cost-effectiveness, and ability to integrate seamlessly with other Azure services. They are triggered by Event Grid to execute both the transcription and summarization tasks, which are written in a Python script in VS Code and deployed to Azure Function Apps, which serves as a container for one or more functions.
• Azure OpenAI Services (Whisper and GPT-4o): This managed service hosts the Whisper model for transcription and GPT-4o for insight extraction and summarization. Whisper offers state-of-the-art accuracy for transcribing audio, including diverse accents and challenging speech patterns, which is critical for high-quality transcriptions. GPT-4o is cost-effective with low latency, ideal for quickly generating concise, actionable summaries.

High-Level implementation workflow

1. Audio File Upload: Users or systems upload audio files to Azure Blob Storage. Blob Storage serves as the starting point and central repository of the pipeline.
2. Event Trigger with Azure Event Grid: Once an audio file is uploaded, Azure Event Grid detects the event and immediately triggers an Azure Function. This real-time processing ensures a seamless automated workflow.
3. Transcription with Whisper: The triggered Azure Function calls the Whisper model through Azure OpenAI Services to transcribe the audio content into text.
4. Insight extraction and summarization with GPT-4o: Once the transcription job is finished, GPT-4o processes the transcribed text to extract the main insights and creates a summary of the presentation. Through effective prompt engineering, the model can be tailored to deliver the desired results. Additionally, GPT-4o offers the ability to format the insights and summarized content in JSON format, enabling easy integration and automated processing.
5. Storage of insights and summaries: The insights and summarized text, as well as the raw transcripts can then be stored in a separate container within Azure Blob Storage for easy access and future reference.

Azure solution

Further extensions

With all the insights stored in a central repository, the pipeline can be expanded to unlock a variety of exciting use cases, as illustrated in the figure below and demonstrate the potential for further expanding how we leverage our collective knowledge to drive innovation.

Possible extensions

Conclusion

Accurate STT technology unlocks a world of possibilities, delivering impactful business outcomes across diverse use cases. In this blog post, we explored how insights derived from audio data can significantly enhance the process of knowledge sharing at ML6. By leveraging Azure’s services, we can transform unstructured audio into searchable, actionable information, making our collective knowledge more accessible and actionable.

Considerations for real-world applications

While our pipeline serves as a solid introduction to Azure and its services, it’s important to consider real-world applications and challenges, where critical factors such as latency and cost come into play. Although Azure Functions are well-suited for event-driven tasks, they have limitations in handling long-running processes due to execution timeout constraints. In such cases, alternative Azure services, like Azure Durable Functions or Azure Container App Jobs, may be more appropriate.

Importance of modularity

Additionally, depending on specific use cases, features like speaker diarization might become increasingly important, for instance, to attribute insights to specific speakers during team discussions. The modular design of our proposed pipeline allows for easy model swapping and customization of workflows based on individual needs. This flexibility makes it convenient to utilize services like Azure Speech Service, which includes built-in speaker diarization and other advanced capabilities.