Day 1: Picking the right Infrastructure

Sovereign AI Builders Log: Day 1

This post is part of my open diary series experimenting with Sovereign AI setup. In the Day 0 post, I laid out the why sovereign AI — avoiding cost linearity, data egress challenges, and a hybrid architecture that won’t burn the wallet while addressing Enterprise priorities. This is a messy, honest reality of Day 1 in the pursuit of selecting the right Infrastructure for this experiment — ambitions, vs. realities, roadblocks, inhibitions and landing on a model infrastructure that finally works.

Day 1 was supposed to be simple. I had already set out the budget for myself for this experiment. An ambitious $1000 max budget, 1–2 hours per weekday and few more additional bit over the weekends. And I had one hard rule that there should be absolutely no credit card surprises. No agents running amok at 2am waking up to a bill that was non-negotiable. What I didn’t expect was spending most of my Day 1 just figuring out where to run the experiment.

Step 1: The hardware Decision

I knew from the start that I wanted to work with medium to large models. My experiment is anchored from the Enterprise context relevance of reverse engineering legacy code, building agentic applications, eventually standing up an internal domain-specific model. Small models weren’t going to cut it.

So, essentially, that left me with four realistic paths:

Path 1— Custom Linux Build

This is definitely the “value king” argument. I was thinking of rebuilding a NVIDIA RTX 3090 that gets me 24GB VRAM, maybe around $750–850 in the market, pairing it with AMD Ryzen 5, 64GB of system RAM, and a decent NVMe drive, approximately costing $1000 all in. Here is where this option killed the utilization math for me. I have 1–2 hours a day, and the machine would sit completely idle or possibly run a few agents for a few more hours, paying for idle hardware drawing power when experimenting felt wrong for this stage.

Path 2 — Mac Mini M4

This was an attractive choice, as it offered the “silent machine” argument. Its unique unified memory architecture means CPU and GPU shared the same memory pool — 24GB would fit a 32B parameter model comfortably. Almost near-silent, ready to use out of the box. The Mac reliability. All attractive options. Two things hit me as I went down this path: 24GB wasn’t shipping until September which means its a non-starter, and a 32GB configuration bumped up the price to M4 Pro at $1400. And again there was the utilization math. So, all put together, I put this pathway in the “revisit if the experiment proves out” pile.

Path 3 — AWS Hosted

I looked at this option seriously, given my AWS background. A g5.xlarge with NVIDIA A10G (24GB VRAM) runs about $1–2/hour on-demand. This is still a hosted option, but nevertheless for enterprises considered as sovereign AI since it will be within their LZ. Here is where I wasn’t comfortable. AWS uses post-paid billing. A runaway agent loop at 2a doesn’t send a warning — it will send me a bill. Over above that, hidden costs for data transfer, IOPS, and public IPv4 makes the final number unpredictable. So, I decided to give this a pass.

Path 4 — Runpod

Runpod offered CPUs, and GPUs running on a prepaid wallet model. I took up spot instances, RTX 3090 of the same configuration was available for $0.44/hour. Runpod certainly matched both my budget constraints and “no bill shock” constraint simultaneously. $20 deposit, offered me roughly 45 hours of runtime on the wallet. Of course, the storage will be persistent and that is something applicable for any private hosted environment as well.

Little more details on Runpod:

Runpod offers a CPU/GPU on-demand environment, and they also have serverless options. As I said, this is nothing different than what other IaaS providers offer, but the thing I liked about these guys was that it was a prepaid wallet. Later in the days, I will talk about what I didn’t like so much as well……

Now to a few basics before you spin off something on runpod — Here is the thing you would need to remember, there are two type of disks on runpod: Container disk, and Volume disk.

Container disk is ephemeral equivalent to the instance storage on an EC2 machine, and the Volume disk is persistent equivalent to the EBS volumes. There is also a network storage option available, but I didn’t think that was required for the experiment at this time. This distinction is important so that you assign storage appropriately to the pod before starting it off. Miss this distinction, and you are downloading a 20GB model at the start of every session. So, always set your ollama models path to /workspace/ollama_models before pulling anything.

Runpod offers multiple templates, and I chose the ollama template on ubuntu configuration: 20GB container disk, 50GB volume disk, roughly costing about $5 per month on storage fees even if stopped.

One final word of warning, if your wallet hits zero and a machine is force-stopped, runpod offers a 96-hour window to add credits before you lose your volume disk.

Step 2: Choosing the Model

I went into this step assuming bigger always meant smarter. I came out of it with a more nuanced view. The two candidates that I seriously considered were Qwen2.5 Coder 32B and a 70B class model. Both are within range for my use cases. Question was whether the capability delta justified the Infrastructure cost.

Here is how my thoughts ran around this problem:

A 70B model at Q4 quantization takes up anywhere around 40-48GB approximately. That’s beyond a single RTX 3090. Running this would mean either moving to a multi-GPU setup or accepting more aggressive quantization. It was a compromise between increased cost for setup or compromising on reasoning quality. Neither felt like the right tradeoff at this stage.

On the other hand, Qwen2.5 Coder 32B lands around 20GB, fitting within my 24GB VRAM leaving headroom for context handling. Also, important to note is that its a code specialized model. 70B alternatives are more general purpose. 32B stood out in coding benchmarks tests and scored competitively against much larger model.

Selected model: Qwen 2.5 Coder 32B Instruct (Q4_K_M quantization). Quantized version fits well inside the RTX 3090 configuration with 24GB VRAM with additional room left for the context as well.

Note to myself: In the future, plan to swap out Qwen and try with Gemma and other equivalents as well…

Step 3: Getting it running…

I wanted to make sure that the ollama model was installed on a custom directory on my workspace volume disk. So, here is what I did:

mkdir -p /workspace/ollama_models
export OLLAMA_MODELS="/workspace/ollama_models"
ollama pull qwen2.5-coder:32b-instruct-q4_K_M

20GB model downloaded in about a minute on the pod. Runpod’s network speed is legitimately fast.

Step 4: Does it actually work?

I didn’t feel like closing Day 1 before a real test. I fed it a chunk of decompiled legacy .NET code — with vague variable names p, q, dr, legacy structures like ArrayList, HashTable, with some hidden business logic in nested conditionals.

I structured the prompt as a proper reverse engineering brief: generate a business requirements document for non-tech stakeholders, identify tech debt and produce a modern typed python equivalent.

What came back was more than I expected:

• Correctly identified that HW in the category code meant “Hardware” inferring from context that was not fed
• Caught that volume discounts and loyalty discounts stack (disc +=0.05 pattern)
• Flagged ArrayList and HashTable as technical debt with reason — boxing/unboxing overheads and type safety issues
• Code was also using MD5 which is a weak hashing algorithm and it was flagged and recommended using SHA-256
• Produced a full pydantic model with validators, type annotations

Verdict: That’s a pass. Legacy .NET reverse engineering is solidly well within the model’s capability.

Where things stand?

Total time spent: roughly 75 minutes; Total spend = $0.24

At this pace, my monthly compute stays under $8 . Storage adds another $5, all under $15/month.

What’s in place:

• Runpod account, prepaid wallet set up for $20, pod stopped to protect balance
• RTX 3090, 50 GB persistent volume at /workspace
• Qwen 2.5 coder 32B downloaded and verified

What’s next?

Currently, the model works in isolation. Day 2 is about making it usable in context getting Cursor on my Laptop, tunneling into the Runpod pod via ngrok, and connecting the two so the model behaves like a local cursor backend. That’s the first step toward something an Enterprise developer could actually use day-to-day.