From Sound to Text: Building a Local AI Assistant with Vosk & PyAudio (Part 2: Full Code)

In Part 1 of this series, we laid the crucial groundwork for our local AI assistant. We equipped it with the ability to hear your voice using Vosk and PyAudio for offline speech-to-text, and to speak its responses using pyttsx3 for offline text-to-speech. You now have a robust, privacy-first audio input and output system running entirely on your machine.

[embed]From Sound to Text: Building a Local AI Assistant with Vosk & PyAudio (Part 1) The idea of a personal AI assistant, ready to answer questions or execute commands, has long captured our imagination…medium.com

However, an assistant that can only echo your words isn’t very intelligent. In this second and final part, we’re going to give our assistant a “brain” and a user-friendly “body.” We’ll integrate a powerful local Large Language Model (LLM) through Ollama to enable intelligent response generation, and build a quick, interactive frontend using Streamlit to orchestrate the entire speech-to-speech experience. Our goal is to create a complete, interactive, local AI chat system that listens, thinks, and responds — all without ever sending your data to the cloud.

Setting Up Your Local LLM Companion

Before we dive into the full code, you’ll need to have Ollama installed and a model downloaded. Ollama makes running large language models locally incredibly straightforward. Ensure the Ollama server is running in the background after installation. Pull a model. For this tutorial, we’re using llama3.2:1b in our code. Now for the exciting part! Below is the complete Python script that integrates all the components we’ve discussed: Vosk for speech-to-text, PyAudio for microphone input, pyttsx3 for text-to-speech, Ollama for intelligent responses, and Streamlit for a user-friendly interface. Create a new Python file (e.g., local_ai_assistant.py) and paste the following code into it:

import streamlit as st
import pyaudio
import json
import time
import pyttsx3
import warnings

# Vosk for local Speech-to-Text
from vosk import Model, KaldiRecognizer

# Langchain components for Ollama LLM integration
from langchain_ollama.llms import OllamaLLM
from langchain_core.prompts import PromptTemplate
from langchain.chains import LLMChain

# Filter out warnings for a cleaner Streamlit output
warnings.filterwarnings("ignore")

# --- Configuration & Initialization ---

# Vosk Model Setup
# Ensure you've downloaded and unzipped your Vosk model into this directory.
# Example: 'vosk-model-en-in-0.5'
MODEL_PATH = "./vosk-model-en-in-0.5"
# Initialize Vosk model and recognizer with the specified sample rate
model = Model(MODEL_PATH)
recognizer = KaldiRecognizer(model, 44100) # Sample rate MUST match PyAudio stream

# Ollama LLM Setup
# Ensure Ollama server is running and the model (e.g., llama3.2:1b) is pulled.
llmmodel = OllamaLLM(model="llama3.2:1b")
# Define the prompt template for our LLM. This gives 'Bob' a persona.
template = """
#### Any prompt for the context to your LLM. ####
User's input: {text}
"""
prompt = PromptTemplate.from_template(template)
# Create a LangChain LLMChain to easily run the prompt through the Ollama model
chain = LLMChain(prompt=prompt, llm=llmmodel)

# pyttsx3 Text-to-Speech Engine Initialization
def initialize_engine():
    """Initializes and configures the pyttsx3 engine for speech output."""
    engine = pyttsx3.init()
    engine.setProperty('rate', 160)   # Speech rate (words per minute)
    engine.setProperty('volume', 0.8) # Volume (0.0 to 1.0)
    voices = engine.getProperty('voices')
    if voices:
        engine.setProperty('voice', voices[0].id) # Use the first available voice
    else:
        print("Warning: No voices found for pyttsx3. Using default system voice.")
    return engine

# pyttsx3 Speak Function
def speak_text(text, engine):
    """Speaks the given text using the initialized pyttsx3 engine."""
    engine.say(text)
    engine.runAndWait()

# PyAudio Setup
p = pyaudio.PyAudio()
# IMPORTANT: Adjust this index to select your microphone.
# You can list devices by running `python -m pyaudio_test` in your terminal.
INPUT_DEVICE_INDEX = 1 # Common for external mics, adjust as needed

def init_stream():
    """Initializes and returns a PyAudio input stream for recording."""
    stream = p.open(format=pyaudio.paInt16,
                    channels=1,
                    rate=44100, # MUST match Vosk recognizer's rate
                    input=True,
                    frames_per_buffer=8192, # Size of audio chunks
                    input_device_index=INPUT_DEVICE_INDEX)
    return stream

def stop_stream(stream):
    """Stops and closes the PyAudio stream."""
    stream.stop_stream()
    stream.close()

# --- Core Listening and Transcription Logic ---
def listen_and_transcribe(tts_engine):
    """
    Listens for user speech via microphone, transcribes it using Vosk,
    and returns the final transcribed text.
    Provides visual feedback in Streamlit.
    """
    stream = init_stream()
    # Read and discard initial audio to flush potential noise/buffer
    time.sleep(0.1) # Small delay
    stream.read(stream.get_read_available(), exception_on_overflow=False)

    st.info("🎙️ Listening... Speak now!")
    # speak_text("Listening", tts_engine) # Optional: have assistant say "listening"

    transcribed_text = ""
    try:
        while True:
            # Read audio data from the stream. exception_on_overflow=False prevents crashes.
            data = stream.read(8192, exception_on_overflow=False)

            # Feed data to Vosk recognizer
            if recognizer.AcceptWaveform(data):
                result = json.loads(recognizer.Result())
                text = result.get("text", "")
                if text:
                    transcribed_text += text + " "
                    # Break after first complete utterance for simplicity in this example
                    # For continuous conversation, this logic would need refinement.
                    break
            ## else:
                # Update partial results in Streamlit for real-time feedback
                ## partial_result = json.loads(recognizer.PartialResult())
                ## if partial_result['partial']:
                    ## st.write(f"Partial: {partial_result['partial']}", ) # Using st.write for simple update
                    # Reset silence timer if any speech is detected (partial or 

    finally:
        stop_stream(stream)
        st.write(f"Final Transcript: {transcribed_text.strip()}")
        # speak_text("Stopped listening", tts_engine) # Optional: confirm listening stopped

    return transcribed_text.strip()

# --- Streamlit UI and Application Flow ---

# Initialize TTS engine once
tts_engine = initialize_engine()

st.set_page_config(page_title="My AI Assistant", layout="centered")
st.title("🎙️ Bob - Your Local AI Assistant")
st.markdown("---")

# Session state to manage UI toggles and persistent data
if "toggle_listen" not in st.session_state:
    st.session_state.toggle_listen = False
if "transcript" not in st.session_state:
    st.session_state.transcript = ""
if "response_text" not in st.session_state:
    st.session_state.response_text = ""

# Button to start/stop listening
if st.button("🎛️ Toggle Listening"):
    st.session_state.toggle_listen = not st.session_state.toggle_listen
    # Clear previous transcript and response when toggling
    st.session_state.transcript = ""
    st.session_state.response_text = ""

# Main logic when listening is toggled on
if st.session_state.toggle_listen:
    st.session_state.transcript = listen_and_transcribe(tts_engine)
    st.session_state.toggle_listen = False # Reset toggle after listening once

# Process transcript if available
if st.session_state.transcript:
    st.success(f"🗣️ You said: {st.session_state.transcript}")

    # Generate LLM response
    with st.spinner("🧠 Bob is thinking..."):
        try:
            # LangChain's chain.run() calls the LLM with the prompt and input
            llm_response = chain.run(text=st.session_state.transcript)
            st.session_state.response_text = llm_response.strip()
        except Exception as e:
            st.error(f"Error generating LLM response: {e}")
            st.session_state.response_text = "I encountered an error while thinking. Please try again."

    # Display and speak the response
    if st.session_state.response_text:
        st.markdown(f"🤖 **Bob says:** {st.session_state.response_text}")
        speak_text(st.session_state.response_text, tts_engine)

    # Clear transcript after processing to avoid re-running LLM on refresh
    st.session_state.transcript = ""

st.markdown("---")
st.markdown("💡 **Tip:** Click 'Toggle Listening' and speak clearly. Bob will respond after you finish your thought.")

# --- End of Code ---

This script orchestrates several powerful tools to create a functional, local AI assistant. Let’s break down its structure and the role of each component.

1. Configuration & Initialization At the top, we set up all our core components. a. Vosk (MODEL_PATH, model, recognizer): These lines initialize the Vosk model from its local path and create the KaldiRecognizer. The 44100 sample rate passed to KaldiRecognizer is critical, as it dictates the audio quality Vosk expects. b. Ollama LLM (llmmodel, template, prompt, chain): We instantiate OllamaLLM to connect to our locally running llama3.2:1b model. The template defines the AI's persona and the structure for prompts. LangChain's LLMChain then bundles this prompt and the LLM for easy execution. c. pyttsx3(initialize_engine, speak_text): These functions manage the text-to-speech engine. initialize_engine configures parameters like speech rate (160 words per minute) and volume (0.8), and attempts to set a specific voice. speak_text queues the text and waits for it to be spoken. d. PyAudio (p, INPUT_DEVICE_INDEX, init_stream, stop_stream): PyAudio is initialized to manage audio input devices. INPUT_DEVICE_INDEX (e.g., 1) is crucial for selecting your specific microphone. The init_stream function configures the audio stream with parameters like format=pyaudio.paInt16 (16-bit audio), channels=1 (mono), rate=44100 (matching Vosk's expectation), and frames_per_buffer=8192 (the size of audio chunks).

2. The Listen and Transcribe Function: Capturing Your Voice This function is the main of our assistant. It starts by initializing a PyAudio stream via init_stream(). A small time.sleep(0.1) and stream.read(..., exception_on_overflow=False)are used to clear any initial buffer noise, ensuring a clean start to listening. st.info("🎙️ Listening... Speak now!") provides real-time feedback in the Streamlit UI. The while True loop continuously reads audio data in chunks (8192 frames) from the microphone. recognizer.AcceptWaveform(data) feeds this audio to Vosk. When Vosk determines a complete utterance has been spoken (often by detecting a pause), it returns True. json.loads(recognizer.Result()) extracts the final recognized text. The loop breaks after the first complete utterance is recognized. Note that recognizer.PartialResult() provides ongoing, real-time recognition updates displayed in Streamlit, showing you what Vosk thinks you're saying as you speak.

3. Streamlit UI and Application Flow This section orchestrates the entire interaction, from listening to responding. a.**tts_engine = initialize_engine(): The pyttsx3 engine is initialized once when the Streamlit app first loads. b. Streamlit Page Configuration: st.set_page_config and st.title set up the web page's appearance. c. `st.session_state:** This is fundamental for Streamlit. It allows variables liketoggle_listen,transcript, andresponse_textto maintain their values across user interactions and page refreshes, which are common in Streamlit apps. d. **Toggle Listening Button:** This button flipsst.session_state.toggle_listen. When True, it triggers thelisten_and_transcribe()function. After transcription, it resets to False, making the process a click-to-listen interaction. e**. Processing the Transcript: **Ifst.session_state.transcriptcontains recognized text, it's displayed to the user (st.success).with st.spinner("🧠 Bob is thinking...")provides a loading indicator. f**. LLM Interaction:**llm_response = chain.run(text=st.session_state.transcript)sends the user's spoken query to our local Ollama LLM, which processes it based on the defined template and generates a response. g. **Display and Speak Response:** The LLM’s response is then displayed in the UI (st.markdown) and spoken aloud usingspeak_text(..., tts_engine). Finally,st.session_state.transcript = ""` clears the transcript, preparing the app for the next interaction.

Conclusion: Your Local AI Assistant is ready !

You’ve done it! You’ve successfully built a complete, end-to-end, local speech-to-speech AI assistant. In this two-part series, we’ve gone from raw audio to intelligent spoken responses, all powered by open-source tools running right on your machine.

Beyond the Basics: This Code as a Launchpad The provided code serves as a robust “head” for a multitude of advanced AI applications. While fully functional, it’s designed to be easily extensible. Consider these avenues for further development:

1. Context-Aware Conversations: Currently, each interaction is independent. To build a truly conversational agent, you would implement a history loop, passing previous messages and responses to the LLM (e.g., using LangChain’s conversational memory features) to maintain context across turns.
2. Enhanced Speech Segmentation: Our current setup listens for a single utterance. For more fluid interaction, you could implement more sophisticated silence detection algorithms or a “wake word” detection system (e.g., using libraries like Porcupine or Snowboy) to continuously listen for commands without explicit button presses.
3. Agentic Capabilities: This assistant can respond, but what if it could act? By integrating tools (e.g., via LangChain agents), your AI could perform web searches, control smart home devices, manage your calendar, or interact with other APIs based on your spoken commands.

This code is a starting point, a powerful template that can be adapted and expanded to fit nearly any local AI voice application you can imagine.

Your Journey into AI Continues!

We hope this series has demystified the process of building a local AI assistant and inspired you to explore the vast potential of AI, ML, and NLP. The world of open-source AI is rapidly evolving, offering incredible opportunities for innovation and personalized solutions.

If you’ve built something cool with this code, or if you have a particular use case or agentic application you’d like to see implemented, let us know in the comments! Your ideas fuel our next explorations.