Custom OpenVINO Server — A Birthday Gift to Myself: How to Celebrate Your 50th Anniversary with the Best Wine — Part 2

Part 1 got some kind attention from the OpenVINO team at Intel — thank you [@Adrian Boguszewski]

If you landed here first, [Part 1 is here ←]

There is this drive inside my head that makes me think I can make things better. That’s exactly why I broke all those toys. But with age and experience I break things less often — and sometimes I actually make them work better.

My attempt to fuel Claude Code with a local server taught me that I shouldn’t try to replace things which are genuinely better and in a league of their own. I shall stay humble. So I did.

I planned to build automatic routing capabilities that would not limit my server to the local GPU only. Humble as every Monday, I started convincing my Claude Code programmer that we could do it.

I explained that I had seen traffic routing once, and that I had already experimented with OVH AI Endpoints. I experienced how easy their service is to use. Is it though? For me — an IT guy with experience — it’s easy. But what about others? This service has one genuinely huge advantage: it complies with European GDPR regulations.

I imagined there are companies — perhaps three — who need exactly that, right?

Somewhere out there, small company staff are stitching together their local inference server with wire and duct tape, just like me, because someone convinced them that Intel Arc B-series GPUs are awesome. People do it every day, right?

To avoid staying locked in my home office in my little bubble with Claude, I decided to take a trip to the office and talk to my friend. He is a pragmatic programmer with a 35-year career. He kindly offered me double coffee and a single face-slap, and put on the table a dozen existing software solutions — and even a photo of a small ARM-powered system from a popular GPU producer.

I told him that I like to play CP2077 on those cards because the volumetric lighting makes Night City neons look awesome — but when I’m not playing CP2077, I don’t like CUDA. I like wine (VINO).

If you’re not Polish like me you might not grasp this little joke immediately, but it might stick. Polish “cuda” means “wonders”.

So I told him: I’ll try to make a fool of myself first and build my server with Intel OpenVINO — and then, if I fail, we can buy whatever BDX thing they sell cheaply for 5k Euro in Poland. Back at home I checked how much money I spent on the Intel Arc B60 — it was less than 5k Euro. How much did I pay in licence fees? Do I need to pay any if I use OpenVINO in a company? Luckily, no.

I mentioned in Part 1 that basic VLM support was there after the second day session — really basic, to the stage where the same model needed to both recognise a picture and chat with the user. If you ask me where I found an OV image of Qwen2.5-VL-7B — I converted it myself. Because I can. Then I found out that on HuggingFace/OpenVINO I can find a faster one.

I didn’t mention earlier that at the model-picking phase I had a fever for converting models. My broadband router needed three cans of cold beer to survive the downloading sessions. Since routers don’t like beer (at least mine doesn’t), I took advantage of the download time. Time for CP2077. Damn — I got rid of that loud RTX monster and now the neons look sh*tty.

Back to OpenVINO — it’s a more interesting and colourful framework for me now. That experiment with model conversion and a little benchmarking taught me that you are generally better off taking a quality pre-converted model from the OpenVINO Team. My models were a bit slower at loading and inference. Perhaps it’s just the fault of my naive approach and incompetence — but here I saw that tinkering doesn’t always give the optimal result.

I started to think I could make something more than just using one model to recognise pictures and another to chat with. What if I could somehow judge which model should be selected dynamically to answer a question?

I asked Claude to spin his shiny silicon bit-brain — and he agreed. We invented something simple together:

Rule signals — regex/heuristic:

• image attached? → VLM
• #code hashtag? → code tier
• system prompt > 4,000 tokens (excluding AnythingLLM tool schemas — more on that below)? → long-context path

Embedding similarity — multilingual-e5-large (already in VRAM for the embeddings endpoint) embeds the query and cosines it against pre-computed task-class centroids. Handles ~75% of unambiguous queries in ~10ms. Threshold: 0.72.

Assessor LLM — qwen3–8b in a dedicated pipeline, asked to pick a task class. Only fires when embedding confidence falls below threshold. ~1s overhead.

I like simple things — until I don’t.

I started testing thoroughly with AnythingLLM. I learned that it injects thousands of tokens of tool schemas into the system prompt, causing every agentic request to trip the long-context gate and route to a document model — evicting whatever was loaded. So we added a fix to count only user and assistant messages toward the token threshold. The system prompt is application boilerplate, not user content.

I stumbled on multiple things during this voyage. A stubborn Python pipeline doesn’t release GPU memory — the C++ destructor only runs at GC collection. Luckily I had Claude, because I’m a destructor myself.

I learned what optimal KV cache size means — the hard way, of course. Two models at 8GB KV each = 29.6GB against 22.71GB total available — the driver was spilling to system RAM silently. Dropped to 3GB KV per model. Lesson learned: the KV cache budget is the single most consequential config number.

I was greedy, stuffing the B60 VRAM like a Thanksgiving turkey. I found out it’s not a turkey. A 20% VRAM reserve — not 5% — is the practical minimum. At 5% free, the continuous batching scheduler runs out of attention computation buffers and generation hangs at 100W, producing nothing.

Step by step, sticking to my mental framework (KYE-SBS-OMK), Claude and I achieved some success. It wasn’t the “holy smart assessor” concept I had imagined when we started — the 8B model wasn’t smart enough and the bigger model was too slow — but it works based on embedding and weight comparison purely. (Sometimes not, though.)

All this experimenting led me to a point where my earlier crafted monitor started to cripple my diagnostic ability. I could see the current state of the working GPU, but I’ll admit — my weak memory didn’t let me hold more than the last 5 seconds of results.

I thought to myself: what can help? Cold beer, CP2077? Perhaps PostgreSQL?

I heard once, at a firepit, when old people were praising some great PostgreSQL ancestor who could do everything better — and they even included the word “vector.” I remember that because “vector” doesn’t fit into a camping landscape very well.

So I discussed with Claude the viability of using PostgreSQL to store live server statistics and provide long-term observability. We also debated storing embedding vectors using the pgvector extension — mainly for providing future data to optimise threshold adjustment for router decisions. That plan went into execution too.

After a short fight with PostgreSQL installation and database structure establishment, we got our first confirmed run with local database observability. Then I noticed the truth in the old saying: the most beautiful painting is useless when you’re blind.

My current monitor was a little Python script and I didn’t want to fiddle more with it — it’s not worth equipping your bicycle with four seats and windshield wipers. Just get a car, I thought.

I discussed options with Claude and we decided to build the lightest possible web interface embedded in the server. Svelte was the front-end framework of my choice because I had some positive experience with it after building a little “Ecosystem Simulator” in another project.

The silicon bit-brain spun multiple times. I criticized, sent it back to the digital workshop bench, and after a couple of feedback rounds we had a good-looking (opinionated) web panel that allowed you to observe server state, loaded models, and historical statistics. The server started to look professional. Some duct tape and loose wire still dangling, though.

SVP dashboard — EnvyStorm ov_server, localhost:11435. The VRAM bar up top tells the most important story: who’s loaded, how hungry they are, and how much headroom is left. Profiles on the right (Fast / Precise / Laborious) map to the router’s model tiers. The Scope switch lets you choose between local-only or local+OVH routing. Manual model selection is available via a dropdown populated with scoped models. The Model Catalogue below lists the whole arsenal — only two soldiers actually on duty at a time. Power draw: 43W idle out of 200W TBP. The GPU is basically napping.

A couple of direct queries from Open WebUI — different models loaded, stats visible. I love watching the CPU core bars dance under pressure.

SVP dashboard ALIVE. Couple direct queries from Open WebUI — different models loaded, stats visible -I love CPU core bar dancing under pressure.

I thought to myself: don’t you feel something is still lacking?

So I built a VRAM profiler a small tool that calculates the exact size of models loaded into VRAM on first load, or manually on demand. Of course PostgreSQL stores this between session, no recalculation needed. That little thing simplified the server’s model eviction decisions considerably. The server now knows up front whether two models with their corresponding KV cache sizes will actually fit on the B60. Goodbye, turkey.

I still had a lot to do. My Google Home speaker — which had seemed so attractive in this new AI era — now looked completely outdated.

How do you make a talking assistant? Something that actually speaks back to you and to Claude? (He revealed his orientation to me.)

And I went to sleep with a head full of questions.

Part 3.

Disclaimer: All opinions, ideas, choices and questionable purchasing decisions described here are exclusively my own. Intel makes good chips. I make trouble and solve some of it for myself. These are separate and independent activities.