Run ONNX Models in .NET: Embeddings and Small Local Models Without Azure

The problem

Three weeks into a semantic search feature, our Azure OpenAI embeddings bill jumped from $180 to $4,300 in a single billing cycle. Nothing in the product had changed. What had changed was a background reindex job that re-embedded every document in the catalog on every deploy, because the “has this changed” check compared the wrong hash field and always came back true. Every deploy re-embedded ~40,000 chunks. Every chunk was a billed API call.

The fix that week was a one-line hash comparison bug. The fix that mattered long-term was different: stop treating an embedding call as something that always has to leave the process. Most of those 40,000 chunks were internal documentation nothing that needed a frontier embedding model, and nothing that needed to touch the network at all.

That’s what pulled me into ONNX Runtime in .NET. Not as a novelty, but as a way to run a small, “good enough” embedding model inside the ASP.NET Core process, with zero per-call cost and no network hop, and reserve the cloud API for the cases that actually need it.

Definition. ONNX (Open Neural Network Exchange) is a standard file format for trained machine learning models. ONNX Runtime is Microsoft’s engine for executing those files — it loads the model’s computation graph and runs inference on CPU (or GPU) without needing Python, PyTorch, or a GPU cluster. In .NET, this means you can load a pre-trained embedding model or small classifier as a .onnx file and run it in-process, using the Microsoft.ML.OnnxRuntime NuGet package.

When to use it. Local ONNX inference makes sense when: the model is small enough to run on CPU in reasonable time (embedding models like MiniLM or BGE-small, not 7B-parameter LLMs), the workload is high-volume and cost-sensitive, or the data can’t leave the network for compliance reasons. It does not make sense when you need frontier-model reasoning quality, when your traffic is low enough that API costs are trivial, or when you don’t want to own model file versioning and hardware sizing yourself.

Why it matters. A support-ticket triage system embedding 50,000 tickets a day at $0.02 per 1,000 tokens adds up fast and most of that traffic is short, routine text where a local MiniLM-class model performs within a few points of a hosted API on retrieval quality. Companies running high-volume internal search, deduplication, or classification increasingly keep a local ONNX model as the default path and call the cloud API only for edge cases.

Concepts overview

1. ONNX and ONNX Runtime — what’s actually happening when you “run a model locally”
2. Getting a pre-trained embedding model into ONNX format
3. Tokenization in .NET with Microsoft.ML.Tokenizers
4. Running inference with InferenceSession
5. Mean pooling and normalization — turning token vectors into one sentence vector
6. Cosine similarity for comparing embeddings
7. Wiring ONNX Runtime into ASP.NET Core dependency injection safely
8. The cost/latency trade-off: deciding when to fall back to a cloud API

1. ONNX and ONNX Runtime

What it is. ONNX is a file format that describes a trained model’s layers, weights, and operations in a framework-neutral way. ONNX Runtime is the execution engine that reads that file and runs the actual matrix multiplications.

Why it exists. Before ONNX, a model trained in PyTorch could really only be served efficiently from a Python process. ONNX decouples training framework from serving framework train in PyTorch or TensorFlow, export once, run anywhere ONNX Runtime is available, including .NET, with no Python dependency in production.

How it works. The exported .onnx file is a graph: nodes are operations (matrix multiply, layer norm, attention), edges are tensors. InferenceSession loads this graph, allocates memory for intermediate tensors, and executes the graph against whatever input tensor you feed it. CPU execution uses vectorized instructions (AVX2/AVX512 where available); no CUDA is required for small models.

ONNX and ONNX Runtime: the same exported file running across runtimes without a Python dependency.

// Partial example — shows the shape of the API, full session covered in Concept 4
using Microsoft.ML.OnnxRuntime;

using var session = new InferenceSession("models/all-MiniLM-L6-v2.onnx");
// session.InputMetadata and session.OutputMetadata describe the graph's
// expected tensor names, shapes, and types — inspect these before wiring
// up your own tensors, since a shape mismatch throws at Run(), not at load.
foreach (var input in session.InputMetadata)
{
    Console.WriteLine($"{input.Key}: {string.Join(",", input.Value.Dimensions)}");
}

2. Getting a pre-trained embedding model into ONNX format

What it is. You rarely train an embedding model yourself — you export an existing one, such as sentence-transformers/all-MiniLM-L6-v2, to the ONNX format using Hugging Face's optimum tooling.

Why it exists. Sentence-transformer models are published as PyTorch checkpoints. ONNX Runtime can’t load a .bin checkpoint directly — it needs the graph-plus-weights .onnx format, plus the tokenizer's vocabulary file.

How it works. This is a one-time, offline step, not something your ASP.NET Core app does at runtime.

getting a model into ONNX format: a one-time offline export, not something the .NET app does at runtime.

# Partial example — run once, offline, not part of the .NET app
pip install optimum[exporters]
optimum-cli export onnx --model sentence-transformers/all-MiniLM-L6-v2 ./exported-model
# Produces: exported-model/model.onnx and exported-model/vocab.txt
# Copy both files into your ASP.NET Core project's content root.

3. Tokenization in .NET with Microsoft.ML.Tokenizers

What it is. Before text reaches the ONNX model, it has to become integer token IDs using the exact same vocabulary the model was trained with. Microsoft.ML.Tokenizers ships a BertTokenizer class that implements WordPiece tokenization the scheme MiniLM and most BERT-family models use.

Why it exists. Get tokenization wrong different casing rules, different special-token handling and the model produces embeddings that are technically valid tensors but semantically meaningless, with no exception telling you so.

How it works. BertTokenizer.Create(vocabPath) loads the vocabulary file exported alongside the model. EncodeToIds converts a string into token IDs, optionally adding the [CLS] and [SEP] special tokens the model expects at the start and end of every sequence.

tokenization: raw text becomes an integer array the model can actually consume.

// Partial example — full pipeline continues in Concept 4
using Microsoft.ML.Tokenizers;

BertTokenizer tokenizer = BertTokenizer.Create("models/vocab.txt");

// addSpecialTokens: true wraps the sequence in [CLS] ... [SEP]
// considerNormalization: true applies lowercasing/accent-stripping to
// match how the model's vocabulary was built
IReadOnlyList<int> tokenIds = tokenizer.EncodeToIds(
    "Reset the customer's password",
    addSpecialTokens: true,
    considerNormalization: true);

Console.WriteLine(string.Join(",", tokenIds));

4. Running inference with InferenceSession

What it is. InferenceSession is the object that actually executes the ONNX graph. You feed it named input tensors and get named output tensors back.

Why it exists. The model graph doesn’t know what “your” input looks like it only knows tensor names and shapes (input_ids, attention_mask, token_type_ids, typically shape [batch, sequence_length]). InferenceSession.Run is the bridge between your C# arrays and the graph's expectations.

How it works. MiniLM’s ONNX export expects three int64 tensors of equal length: token IDs, an attention mask (1 for real tokens, 0 for padding), and token type IDs (0 for single-sequence input). The output is a [batch, sequence_length, 384] tensor — one 384-dimensional vector per token, not yet a single sentence vector.

running inference: three named tensors go in, Run() executes the graph, one tensor comes out.

// Partial example — pooling into a single vector is Concept 5
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

int[] ids = tokenIds.ToArray();
int seqLen = ids.Length;

var inputIdsTensor = new DenseTensor<long>(new[] { 1, seqLen });
var attentionMaskTensor = new DenseTensor<long>(new[] { 1, seqLen });
var tokenTypeTensor = new DenseTensor<long>(new[] { 1, seqLen });

for (int i = 0; i < seqLen; i++)
{
    inputIdsTensor[0, i] = ids[i];
    attentionMaskTensor[0, i] = 1;   // no padding in a single-sequence batch of 1
    tokenTypeTensor[0, i] = 0;       // single sequence, not a sentence pair
}

var inputs = new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("input_ids", inputIdsTensor),
    NamedOnnxValue.CreateFromTensor("attention_mask", attentionMaskTensor),
    NamedOnnxValue.CreateFromTensor("token_type_ids", tokenTypeTensor)
};

using var results = session.Run(inputs);
var tokenEmbeddings = results.First(r => r.Name == "last_hidden_state").AsTensor<float>();

5. Mean pooling and normalization

What it is. Sentence-transformer models like MiniLM are trained so that averaging their per-token output vectors weighted by the attention mask, to ignore padding produces a meaningful sentence-level embedding.

Why it exists. The raw model output is one vector per token. Search and similarity use cases need one vector per document. Mean pooling is the specific aggregation MiniLM was trained to support; other aggregations (like just taking the [CLS] token vector) give worse results for this model family.

How it works. For each dimension, average across all real tokens, excluding padding positions. Then L2-normalize the result so cosine similarity reduces to a dot product.

mean pooling and normalization: per-token vectors collapse into one sentence vector, ignoring padding.

// Partial example — assumes tokenEmbeddings and attentionMaskTensor from Concept 4
static float[] MeanPoolAndNormalize(Tensor<float> tokenEmbeddings, long[] attentionMask)
{
    int seqLen = tokenEmbeddings.Dimensions[1];
    int hiddenSize = tokenEmbeddings.Dimensions[2];
    var pooled = new float[hiddenSize];
    int realTokenCount = 0;

    for (int t = 0; t < seqLen; t++)
    {
        if (attentionMask[t] == 0) continue; // skip padding positions
        realTokenCount++;
        for (int d = 0; d < hiddenSize; d++)
            pooled[d] += tokenEmbeddings[0, t, d];
    }

    for (int d = 0; d < hiddenSize; d++)
        pooled[d] /= realTokenCount;

    // L2 normalize so cosine similarity becomes a plain dot product downstream
    float norm = MathF.Sqrt(pooled.Sum(v => v * v));
    for (int d = 0; d < hiddenSize; d++)
        pooled[d] /= norm;

    return pooled;
}

6. Cosine similarity for comparing embeddings

What it is. A similarity score between two embeddings, ranging from -1 (opposite meaning) to 1 (identical meaning), computed as the dot product of two L2-normalized vectors.

Why it exists. Embeddings are only useful relative to each other. Semantic search, deduplication, and clustering all reduce to “which stored vectors are closest to this query vector.”

How it works. Because Concept 5 already normalized both vectors, similarity is just a dot product — no square roots needed at comparison time, which matters when you’re comparing one query against thousands of stored vectors.

cosine similarity: the angle between two normalized vectors, read off as a score.

// Partial example
static float CosineSimilarity(float[] a, float[] b)
{
    float dot = 0f;
    for (int i = 0; i < a.Length; i++)
        dot += a[i] * b[i];
    return dot; // vectors are already unit-normalized, so this IS the cosine similarity
}

7. Wiring ONNX Runtime into ASP.NET Core safely

What it is. InferenceSession is expensive to construct (it loads and validates the whole graph) but its Run method is documented as safe for concurrent calls. That makes it a singleton, not a scoped or transient service.

Why it exists. Registering InferenceSession as scoped means a new multi-hundred-millisecond load on every HTTP request. Registering it as singleton but then mutating shared state inside it (which the hands-on section below has to actively avoid) reintroduces the concurrency bug that singletons are supposed to sidestep.

How it works. Load the session once at startup, register it as a singleton, and keep the tokenizer as a singleton too since BertTokenizer holds no per-request state.

DI wiring: one model load at startup, many concurrent requests sharing the same singleton session.

// Partial example — full DI wiring for the hands-on service is in the next section
builder.Services.AddSingleton<InferenceSession>(_ =>
    new InferenceSession(Path.Combine(builder.Environment.ContentRootPath, "Models", "all-MiniLM-L6-v2.onnx")));

builder.Services.AddSingleton<BertTokenizer>(_ =>
    BertTokenizer.Create(Path.Combine(builder.Environment.ContentRootPath, "Models", "vocab.txt")));

8. The cost/latency trade-off: when to fall back to a cloud API

What it is. A local MiniLM-class model is free per call and typically 5–20ms on CPU for short text, but it has a fixed context window (512 tokens for MiniLM) and lower ceiling on embedding quality than a large hosted model.

Why it exists. Not every input fits the local model’s constraints. Long documents, non-English text the local model wasn’t trained on, or cases where you need the highest achievable retrieval quality still justify a cloud call — the point isn’t to eliminate the cloud API, it’s to stop paying for it on every call by default.

How it works. The hands-on build below implements this as an explicit decision point: check length/eligibility first, use the free local path when eligible, and only reach for the metered cloud path — with safeguards against the concurrency trap that caused the original cost spike — when it isn’t.

Hands-on: a hybrid embedding service with a local-first, cloud-fallback path

Problem statement. We’re going to build an ASP.NET Core Web API that generates text embeddings for a search feature. Most incoming text is short (support tickets, chat messages, doc chunks under 512 tokens) and should be embedded locally for free. Text that exceeds the local model’s token limit should fall back to a cloud embedding API. Under concurrent load, repeated identical requests for the same text must not each trigger a separate cloud call — that thundering-herd pattern is exactly the shape of cost spike from the opening.

Step 1 — Project setup

dotnet new webapi -n LocalEmbeddings.Api -controllers
cd LocalEmbeddings.Api

dotnet add package Microsoft.ML.OnnxRuntime --version 1.28.0
dotnet add package Microsoft.ML.Tokenizers --version 2.0.0
dotnet add package Microsoft.Extensions.AI --version 9.4.0
dotnet add package Microsoft.Extensions.AI.OpenAI --version 9.4.0-preview.1.25207.5
dotnet add package Microsoft.Extensions.Caching.Memory --version 9.0.0

Download the exported model files from Step 2 of the Concepts section above (model.onnx renamed to all-MiniLM-L6-v2.onnx, and vocab.txt) and place them under Models/ in the project, then mark them "Copy to Output Directory: Copy if newer" in the .csproj:

<ItemGroup>
  <None Include="Models\all-MiniLM-L6-v2.onnx">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
  <None Include="Models\vocab.txt">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
</ItemGroup>

Step 2 — Define the shared types (Models/EmbeddingModels.cs)

Every type used below is defined here, in the hands-on section — none of it depends on classes from the concept walkthrough above.

namespace LocalEmbeddings.Api.Models;

public sealed record EmbeddingResult(float[] Vector, string Source, int TokenCount);

public sealed record EmbeddingRequest(string Text);

public sealed class EmbeddingOptions
{
    // MiniLM's exported graph truncates/errors above this sequence length;
    // anything longer routes to the cloud provider instead of failing.
    public int MaxLocalTokens { get; set; } = 256;
}

Step 3 — The local ONNX provider (Services/LocalOnnxEmbedder.cs)

This is the non-trivial decision point for the local path: enforcing the token-length boundary explicitly, rather than letting the ONNX graph throw a shape-mismatch exception on oversized input.

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using Microsoft.ML.Tokenizers;
using LocalEmbeddings.Api.Models;

namespace LocalEmbeddings.Api.Services;

public interface ILocalEmbedder
{
    // Returns null when the input exceeds what the local model can safely handle,
    // signaling the caller to use the cloud fallback instead of failing the request.
    EmbeddingResult? TryEmbed(string text, int maxTokens);
}

public sealed class LocalOnnxEmbedder : ILocalEmbedder
{
    private readonly InferenceSession _session;
    private readonly BertTokenizer _tokenizer;

    public LocalOnnxEmbedder(InferenceSession session, BertTokenizer tokenizer)
    {
        _session = session;
        _tokenizer = tokenizer;
    }

    public EmbeddingResult? TryEmbed(string text, int maxTokens)
    {
        var tokenIds = _tokenizer.EncodeToIds(text, addSpecialTokens: true, considerNormalization: true);

        if (tokenIds.Count > maxTokens)
            return null; // caller falls back to cloud rather than truncating silently

        int seqLen = tokenIds.Count;
        var inputIds = new DenseTensor<long>(new[] { 1, seqLen });
        var attentionMask = new DenseTensor<long>(new[] { 1, seqLen });
        var tokenTypeIds = new DenseTensor<long>(new[] { 1, seqLen });

        for (int i = 0; i < seqLen; i++)
        {
            inputIds[0, i] = tokenIds[i];
            attentionMask[0, i] = 1;
            tokenTypeIds[0, i] = 0;
        }

        var inputs = new List<NamedOnnxValue>
        {
            NamedOnnxValue.CreateFromTensor("input_ids", inputIds),
            NamedOnnxValue.CreateFromTensor("attention_mask", attentionMask),
            NamedOnnxValue.CreateFromTensor("token_type_ids", tokenTypeIds)
        };

        // InferenceSession.Run is safe to call concurrently from multiple requests;
        // no lock needed here as long as the session is registered as a singleton
        // and no request mutates the session itself.
        using var results = _session.Run(inputs);
        var tokenEmbeddings = results.First(r => r.Name == "last_hidden_state").AsTensor<float>();

        var pooled = MeanPoolAndNormalize(tokenEmbeddings, attentionMask);
        return new EmbeddingResult(pooled, "local-onnx", seqLen);
    }

    private static float[] MeanPoolAndNormalize(Tensor<float> tokenEmbeddings, DenseTensor<long> attentionMask)
    {
        int seqLen = tokenEmbeddings.Dimensions[1];
        int hiddenSize = tokenEmbeddings.Dimensions[2];
        var pooled = new float[hiddenSize];
        int realTokenCount = 0;

        for (int t = 0; t < seqLen; t++)
        {
            if (attentionMask[0, t] == 0) continue;
            realTokenCount++;
            for (int d = 0; d < hiddenSize; d++)
                pooled[d] += tokenEmbeddings[0, t, d];
        }

        for (int d = 0; d < hiddenSize; d++)
            pooled[d] /= realTokenCount;

        float norm = MathF.Sqrt(pooled.Sum(v => v * v));
        for (int d = 0; d < hiddenSize; d++)
            pooled[d] /= norm;

        return pooled;
    }
}

Step 4 — The cloud fallback provider (Services/CloudEmbedder.cs)

using Microsoft.Extensions.AI;
using LocalEmbeddings.Api.Models;

namespace LocalEmbeddings.Api.Services;

public interface ICloudEmbedder
{
    Task<EmbeddingResult> EmbedAsync(string text, CancellationToken ct);
}

public sealed class CloudEmbedder : ICloudEmbedder
{
    private readonly IEmbeddingGenerator<string, Embedding<float>> _generator;

    public CloudEmbedder(IEmbeddingGenerator<string, Embedding<float>> generator)
    {
        _generator = generator;
    }

    public async Task<EmbeddingResult> EmbedAsync(string text, CancellationToken ct)
    {
        var embedding = await _generator.GenerateAsync(new[] { text }, cancellationToken: ct);
        return new EmbeddingResult(embedding[0].Vector.ToArray(), "cloud-fallback", TokenCount: -1);
    }
}

Step 5 — The hybrid gateway with thundering-herd protection (Services/HybridEmbeddingService.cs)

This is the concurrency decision that ties the whole POC together. A naive cache — check dictionary, miss, call cloud, store result — lets N concurrent requests for the same uncached text all miss simultaneously and all fire a cloud call. GetOrAdd with a Lazy<Task<T>> collapses that down to exactly one in-flight cloud call per key.

using System.Collections.Concurrent;
using Microsoft.Extensions.Options;
using LocalEmbeddings.Api.Models;

namespace LocalEmbeddings.Api.Services;

public sealed class HybridEmbeddingService
{
    private readonly ILocalEmbedder _local;
    private readonly ICloudEmbedder _cloud;
    private readonly int _maxLocalTokens;

    // Keyed on the input text. Lazy<Task<T>> ensures that even if 50 requests
    // arrive for the same uncached text at once, only the first constructs
    // the Task (and therefore only the first calls the cloud API) — the other
    // 49 await the same in-flight Task instead of starting their own.
    private readonly ConcurrentDictionary<string, Lazy<Task<EmbeddingResult>>> _cache = new();

    public HybridEmbeddingService(ILocalEmbedder local, ICloudEmbedder cloud, IOptions<EmbeddingOptions> options)
    {
        _local = local;
        _cloud = cloud;
        _maxLocalTokens = options.Value.MaxLocalTokens;
    }

    public Task<EmbeddingResult> GetEmbeddingAsync(string text, CancellationToken ct)
    {
        var lazy = _cache.GetOrAdd(text, key => new Lazy<Task<EmbeddingResult>>(
            () => ResolveAsync(key, ct),
            LazyThreadSafetyMode.ExecutionAndPublication));

        return lazy.Value;
    }

    private async Task<EmbeddingResult> ResolveAsync(string text, CancellationToken ct)
    {
        var local = _local.TryEmbed(text, _maxLocalTokens);
        if (local is not null)
            return local;

        // Only text that exceeds the local model's token budget reaches here —
        // this is the deliberately narrow, metered path.
        return await _cloud.EmbedAsync(text, ct);
    }
}

Step 6 — The controller (Controllers/EmbeddingsController.cs)

Controller-based, not minimal API, because HybridEmbeddingService has three constructor-injected dependencies and a clear single responsibility — DI wiring stays explicit and testable.

using Microsoft.AspNetCore.Mvc;
using LocalEmbeddings.Api.Models;
using LocalEmbeddings.Api.Services;

namespace LocalEmbeddings.Api.Controllers;

[ApiController]
[Route("api/[controller]")]
public sealed class EmbeddingsController : ControllerBase
{
    private readonly HybridEmbeddingService _hybrid;

    public EmbeddingsController(HybridEmbeddingService hybrid)
    {
        _hybrid = hybrid;
    }

    [HttpPost]
    public async Task<ActionResult<EmbeddingResult>> Generate(
        [FromBody] EmbeddingRequest request, CancellationToken ct)
    {
        if (string.IsNullOrWhiteSpace(request.Text))
            return BadRequest("Text cannot be empty.");

        var result = await _hybrid.GetEmbeddingAsync(request.Text, ct);
        return Ok(result);
    }
}

Step 7 — Wiring it all up (Program.cs)

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.Tokenizers;
using Microsoft.Extensions.AI;
using LocalEmbeddings.Api.Models;
using LocalEmbeddings.Api.Services;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();
builder.Services.Configure<EmbeddingOptions>(builder.Configuration.GetSection("Embedding"));

// Singleton: expensive to construct, safe to share across concurrent requests
builder.Services.AddSingleton(_ =>
    new InferenceSession(Path.Combine(builder.Environment.ContentRootPath, "Models", "all-MiniLM-L6-v2.onnx")));

builder.Services.AddSingleton(_ =>
    BertTokenizer.Create(Path.Combine(builder.Environment.ContentRootPath, "Models", "vocab.txt")));

builder.Services.AddSingleton<ILocalEmbedder, LocalOnnxEmbedder>();

// Cloud fallback provider — reads endpoint/key from configuration, never hardcoded
builder.Services.AddSingleton<IEmbeddingGenerator<string, Embedding<float>>>(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    var endpoint = config["AzureOpenAI:Endpoint"]
        ?? throw new InvalidOperationException("AzureOpenAI:Endpoint is not configured.");
    var deployment = config["AzureOpenAI:EmbeddingDeployment"] ?? "text-embedding-3-small";

    return new Azure.AI.OpenAI.AzureOpenAIClient(
            new Uri(endpoint),
            new Azure.Identity.DefaultAzureCredential())
        .GetEmbeddingClient(deployment)
        .AsIEmbeddingGenerator();
});

builder.Services.AddSingleton<ICloudEmbedder, CloudEmbedder>();
builder.Services.AddSingleton<HybridEmbeddingService>();

var app = builder.Build();
app.MapControllers();
app.Run();

Add the cloud provider’s endpoint to appsettings.json (using Key Vault / Managed Identity for the actual secret in production, as covered in the earlier security post in this series):

{
  "Embedding": { "MaxLocalTokens": 256 },
  "AzureOpenAI": {
    "Endpoint": "https://your-resource.openai.azure.com/",
    "EmbeddingDeployment": "text-embedding-3-small"
  }
}

Final verification

Run the app, then send a short request:

curl -X POST https://localhost:5001/api/Embeddings \
  -H "Content-Type: application/json" \
  -d '{"text":"Reset the customer'\''s password"}'

Expected output — "source":"local-onnx", a 384-length vector array, and a tokenCount under 256:

{ "vector": [0.0123, -0.0456, ...], "source": "local-onnx", "tokenCount": 9 }

To confirm the thundering-herd protection, fire 20 concurrent identical requests for a long piece of text (over 256 tokens, forcing the cloud path) and log a counter inside CloudEmbedder.EmbedAsync. The counter should increment exactly once, not 20 times — the other 19 requests resolve from the same in-flight Task.

Source code:

https://github.com/bhargavkoya/AI_Dotnet_Integration_Engineer_Learnings/tree/9896e3de2ba87de0fdfa14e08c126f0edd7f4ce9/LocalEmbeddings

Closing

Best practices:

• Register InferenceSession and the tokenizer as singletons; never scope them per request.
• Enforce the model’s token limit explicitly in code rather than letting a shape mismatch throw from inside Run().
• L2-normalize embeddings once, at generation time, not on every similarity comparison.
• Version your .onnx and vocab.txt files together — swapping one without the other produces embeddings that are wrong without erroring.
• Guard any cache-miss path that calls a metered API with a mechanism (like Lazy<Task<T>>) that collapses concurrent misses into one call.

Common mistakes:

• Registering InferenceSession as scoped or transient, paying the model-load cost on every request catch it by profiling first-request latency in load tests.
• Averaging in padding tokens during mean pooling, silently degrading every embedding — catch it by unit-testing pooling output against a known reference vector from the Python sentence-transformers library.
• Assuming EncodeToIds and the model's expected sequence length always agree catch it by asserting tokenIds.Count <= maxTokens before building tensors, not after Run() throws.

Bottlenecks at scale. CPU-bound InferenceSession.Run calls will saturate available cores under high concurrency; teams handle this by batching multiple texts into a single Run call (shape [batch, seq_len] instead of [1, seq_len]) rather than spinning up one session per request, and by moving to GPU-backed Microsoft.ML.OnnxRuntime.Gpu only once CPU throughput is measured and found insufficient — not by default.

Pros and cons. Pros: zero marginal cost per call, no network round-trip, no compliance exposure from sending data off-network. Cons: lower embedding quality ceiling than large hosted models, CPU capacity planning becomes your problem instead of a cloud provider’s, and you own model file versioning across deploys.

Alternatives:

• ML.NET’s own embedding pipelines — simpler API surface, but less flexibility over pooling strategy and tokenizer choice than raw ONNX Runtime.
• Always calling a cloud embedding API — simplest to build, but reintroduces exactly the cost-per-call exposure this post exists to avoid.
• A dedicated embedding microservice in Python (FastAPI + sentence-transformers) — access to the full sentence-transformers ecosystem, at the cost of an extra network hop and a second runtime to operate.

Four years into running .NET services in production, the pattern I keep coming back to is this: the expensive path should be the exception you explicitly route to, not the default you fall into by not thinking about it. That reindex bug wasn’t really a hashing bug — it was a system with no cheap default. Worth checking: does your codebase have a call path today where “just call the API” is the only option, even for the 90% of traffic that doesn’t need it?