Transcription That Also Hears Emotion: A Local Speech-Understanding API with SenseVoice and FastAPI

Most speech-to-text pipelines throw the interesting part away. You get the words, and everything else — was the speaker angry, was there laughter on the line — is gone. The usual fix is to bolt a second model onto the first: one for transcription, another for sentiment. That is two models to load and reconcile.

The objective of this exercise was to get the words, the emotion, and the audio events out of a single pass. SenseVoice does exactly that: automatic speech recognition, speech emotion recognition. We wrapped it in a FastAPI endpoint that takes an audio file and returns structured JSON, running locally on Apple Silicon — no API keys, no audio leaving the machine. Below are the steps and the code.

Step 1 — Virtual environment

Seven dependencies, listed in requirements.txt. funasr pulls the model framework; the iic/SenseVoiceSmall weights download from ModelScope on first run, so there is no separate download step. python-multipart is what lets FastAPI accept the file upload. fastapi uvicorn funasr modelscope torch torchaudio python-multipart

import os
import re
import shutil
import tempfile
from fastapi import FastAPI, UploadFile, File, HTTPException
from funasr import AutoModel

Step 2 — Load the model once

The model is loaded outside the route, at module import, so it initializes a single time when the server boots rather than on every request. SenseVoiceSmall is small enough to sit in memory comfortably.

import os
import re
import shutil
import tempfile
from fastapi import FastAPI, UploadFile, File, HTTPException
from funasr import AutoModel

app = FastAPI(title="End-to-End Emotion Detection API")

# --- Model Initialization ---
# We load the model outside the route so it only initializes once on startup.
print("Loading SenseVoice model into memory...")
model = AutoModel(
    model="iic/SenseVoiceSmall",
    trust_remote_code=True,
    device="cpu"  # For Apple Silicon, "cpu" is highly stable out of the box. You can experiment with "mps" for GPU acceleration.
)
print("Model ready.")

Step 3 — The API

The whole service is one file. A POST endpoint validates the file type, writes the upload to a temp file (the model reads from a path, not a buffer), runs inference, parses the result, and cleans up the temp file in a finally block so nothing leaks to disk.

import os
import re
import shutil
import tempfile
from fastapi import FastAPI, UploadFile, File, HTTPException
from funasr import AutoModel

app = FastAPI(title="End-to-End Emotion Detection API")

# --- Model Initialization ---
# We load the model outside the route so it only initializes once on startup.
print("Loading SenseVoice model into memory...")
model = AutoModel(
    model="iic/SenseVoiceSmall",
    trust_remote_code=True,
    device="cpu"  # For Apple Silicon, "cpu" is highly stable out of the box. You can experiment with "mps" for GPU acceleration.
)
print("Model ready.")

@app.post("/detect-emotion")
async def detect_emotion(audio_file: UploadFile = File(...)):
    if not audio_file.filename.endswith(('.wav', '.mp3', '.m4a')):
        raise HTTPException(status_code=400, detail="Invalid file type. Please upload an audio file (.wav, .mp3, .m4a).")

    # 1. Save the uploaded file to disk temporarily for the model to read
    with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as temp_audio:
        shutil.copyfileobj(audio_file.file, temp_audio)
        temp_file_path = temp_audio.name

    try:
        # 2. Run Inference
        result = model.generate(
            input=temp_file_path,
            cache={},
            language="auto",
            use_itn=True
        )

        raw_output = result[0]["text"]

        # 3. Parse the Rich Transcription
        emotion_match = re.search(r'<\|(HAPPY|SAD|ANGRY|NEUTRAL|SURPRISED|FEAR)\|>', raw_output, re.IGNORECASE)
        emotion = emotion_match.group(1).upper() if emotion_match else "NEUTRAL"

        # Parse audio events
        event_match = re.findall(r'<\|(Laughter|Applause|Music|Cough|Sneeze|Crying)\|>', raw_output, re.IGNORECASE)
        events = [e.capitalize() for e in event_match] if event_match else []

        # Remove all the bracketed tags to get just the spoken text
        clean_transcript = re.sub(r'<\|.*?\|>', '', raw_output).strip()

        # 4. Return the structured JSON payload
        return {
            "status": "success",
            "emotion": emotion,
            "events": events,
            "transcript": clean_transcript,
            "raw_output": raw_output
        }

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

    finally:
        # 5. Cleanup the temporary file to prevent storage leaks
        if os.path.exists(temp_file_path):
            os.remove(temp_file_path)

Step 4 — Parsing the rich transcription

SenseVoice does not hand back tidy fields. It returns one string with everything encoded as inline tags at the front:

<|en|><|ANGRY|><|Speech|>I have called three times about this charge and nobody has fixed it.

So the parsing is three small regexes against that string. Pull the emotion tag, defaulting to NEUTRAL when none is present. Collect any audio-event tags — laughter, applause, music, cough, sneeze, crying.

The raw output goes back in the payload too, so nothing the model said is lost if you want to parse more later.

Two inference flags shape the text: language=”auto” lets the model detect the language, and use_itn=True turns on inverse text normalization, so numbers and punctuation come out readable

Step 5 — Run it

Start the server with uvicorn, then send it an audio file with curl. The repo includes a customer_call.wav to try it against.

# Install and launch the API 
pip install -r requirements.txt 
uvicorn main:app -- host 0.0.0.0 -- port 8000 

# In another terminal, send it an audio file 

curl -X POST "http://127.0.0.1:8000/detect-emotion" \ -F "audio_file=@customer_call.wav"

The response is as below:

curl -s -X POST "http://127.0.0.1:8000/detect-emotion" -H "accept: application/json" -H "Content-Type: multipart/form-data" -F "audio_file=@/Users/vk/Downloads/anger.wav"
{"status":"success","emotion":"ANGRY","events":[],"transcript":"What can you prove it, what proof do you have, can you prove it?","raw_output":"<|en|><|ANGRY|><|Speech|><|withitn|>What can you prove it, what proof do you have, can you prove it?"}%                                                                               vk@Varuns-Air trascribe_with_emotions %

Outcome

One model, one inference pass, three signals: the transcript, the speaker’s emotion, and any audio events — behind a single endpoint that any other tool can POST a file to.

The split that usually costs you two models collapses into about sixty lines and one set of tags to parse.

The next steps for the experiment are the obvious ones: try the “mps” device, batch multiple files per request, and pull SenseVoice’s timestamp output through so emotion can be tracked across a long call rather than summarized into a single label.

The emotion label is a coarse, whole-clip read — useful for triage, not a substitute for listening.

As always, create a separate virtual environment for testing.