AI Privacy for Small Shops

I Blasted Grok’s ZDR Problem. Then I Realized They Were the Only Ones Giving Me a Switch

OpenAI and Anthropic may have enterprise answers. Local models may have promise. But for a small shop using coding agents today, Grok gave me the closest thing to a usable privacy control.

Over the July 17–18 weekend, I was listening to the All-In Podcast when the conversation turned to Grok Build, data retention, and the question that has been sitting in the back of my mind for a while:

How exposed am I when I point an AI coding agent at my code?

That is not a theoretical question for me.

I run a small software consulting company. My work involves private source code, infrastructure automation, internal systems, and customer environments that I cannot casually feed into whatever tool happens to have the best demo that week.

At the same time, I do not want to pretend AI coding agents are optional anymore. They have changed how I build software. The question is no longer whether I should use them.

The question is how to use them without giving away more control than I intended.

That weekend I decided to run a real experiment.

Could I keep the coding-agent experience I liked, but move the model work onto infrastructure I controlled?

In plain English:

Could I use Grok Build as the front end, but point it at my own local LLMs?

My first reaction to the Grok retention story was pretty simple: this is exactly the kind of thing developers should be upset about. If a coding agent is touching whole repositories, the retention path cannot be vague. That deserves scrutiny.

But after spending the weekend trying to get away from cloud coding agents entirely, I ended up in a more complicated place.

Grok had a problem worth criticizing.

But Grok also gave me something OpenAI, Anthropic, and local models could not give me in a practical way: a ZDR switch I could actually turn on.

Why This Matters To Me

AI coding agents are becoming part of the engineering process, so the privacy path matters.

I have spent most of my career integrating systems that were not designed to work together.

That was true when I worked on traffic control systems in the 1980s. It was true when I helped build enterprise integration systems. It is still true today with Kubernetes, AWS, CI/CD pipelines, and AI agents.

The tools change. The underlying problem usually does not.

You have a set of useful systems. Each one does something important. The real value shows up when you connect them into a process that is more useful than the individual pieces.

AI coding agents are another version of that problem.

The agent is not just a chatbot. It reads files. It edits files. It runs commands. It carries context. It learns project rules. It starts to behave like another participant in the engineering process.

That is powerful.

It is also why privacy matters.

If an AI agent can understand enough of my repository to make useful changes, then it can also expose enough of my repository to create a real business problem if the retention path is wrong.

That is the tension I was trying to resolve.

I wanted the best coding-agent workflow I could get, but I wanted the model side of the work to stay under my control as much as possible.

Alex Karp made a related point in a CNBC interview around the same time. His language was bigger-company language, but the part that stuck with me was the idea that technical customers want to own the “means of production.”

I am not Palantir’s customer base. I am a small shop.

That makes the problem more important, not less. I do not have a legal department sitting behind every experiment. If my playbooks, repository structure, customer patterns, or billing workflows leak into the wrong retention path, that is not an abstract governance issue. That is my business.

The Starting Point

I wanted Grok Build’s agent experience without treating cloud Grok as non-negotiable.

The thing that caught my attention was that Grok Build supports custom models.

The official Grok Build docs describe it as a coding agent that can run in an interactive terminal UI or headlessly, and they also document custom model configuration through ~/.grok/config.toml.

That matters.

It means Grok Build does not have to be only “Grok in the cloud or nothing.” In theory, I could treat Grok Build as the agent shell and point it at an OpenAI-compatible endpoint running locally.

That was the experiment.

I had already tried a few other agent front ends.

I tried Aider. I liked parts of it, but my real work often spans many interconnected repositories. One playbook calls many roles. One repo points to several others. I do not always have the luxury of treating a task as a single-repo edit.

I also tried OpenCode. It was interesting, but it did not feel as mature for the way I work.

Grok Build felt closer to the kind of professional agent experience I had become used to with Claude Code: a serious terminal UI, repo awareness, tool execution, and a workflow that felt like it could support real engineering work.

So the question became simple:

Can I keep that front end and swap out the brain?

The Local Stack

The first answer was simple on paper: Grok Build in front, local inference in the back.

The first version looked like this:

Grok Build
    -> LiteLLM on localhost
        -> Ollama on localhost
            -> qwen3:8b, llama3.1:8b, mistral-nemo, qwen2.5-coder, ...

Grok Build was the agent shell.

LiteLLM was the local OpenAI-compatible router / orchestrator.

Ollama served the local models.

The idea was clean. Grok Build would talk to localhost. LiteLLM would route the request. Ollama would run the model. Customer code and prompts would not need to leave the machine for normal coding work.

I also tried bypassing LiteLLM and pointing Grok Build directly at Ollama’s OpenAI-compatible endpoint.

That ended up mattering.

LiteLLM is useful, but the router had issues. In my test, the chat path could look fine while the agent path was broken. Tool calls are where the truth shows up. At one point, streaming tool calls were getting mangled into JSON-as-content. The model appeared to be talking, but the agent harness was no longer getting clean tool-call behavior.

For the local agent path that worked best, I ended up trusting qwen3:8b through direct Ollama more than the routed setup.

That was the first practical lesson:

Do not test local agents by asking whether chat works. Test whether tools work.

Porting The Agent’s Memory

A coding agent is only useful if it carries the rules and habits of the way I actually work.

I did not want a clean Grok install with amnesia.

Claude Code had months of project memory built around how I work: shell hygiene, bookkeeping rules, what directories not to write into, how to audit file edits, how to handle internet fetches, and what kinds of commands should ask before running.

That memory is not a nice-to-have. It is part of the engineering system.

So I copied the important parts into Grok-native locations:

Existing practiceGrok-side equivalentGlobal shell and safety rules~/.grok/rules/Project memory and known failure modes~/.grok/memory/imported/File edit audit habits~/.local/state/grok-agent/Repeatable proceduresGrok skillsPermission instinctsGrok deny / ask lists

That part worked, and I still think it matters.

The lesson is that you do not just move prompts between tools. You move institutional memory.

If a coding agent is going to become part of how you build software, then the rules, scars, and habits around the work need to be portable.

The Login Problem

If cloud Grok is still in the model catalog, the system is not truly local.

Here is the part I did not expect to spend so much time on.

Even when I routed inference to LiteLLM or Ollama, stock Grok Build still wanted an xAI login for long stretches of the experiment.

At first, that was confusing.

It was tempting to think:

If Grok Build is still asking me to log in, then maybe my prompts are still going to xAI.

But that was not exactly what was happening.

When the selected model’s base_url was loopback, the model completions for that path were local. LiteLLM talked to Ollama. Ollama ran the model. The tokens were not coming from api.x.ai.

The problem was more subtle.

Grok Build still had cloud Grok available as an option in the model catalog. Sometimes it was the default. Sometimes it was available as a secondary or fallback model. Sometimes web-search or fork-style behavior still referenced a cloud Grok model.

That meant I was not really in a local-only product.

I was in a logged-in Grok product that also happened to know about localhost.

That distinction matters.

The breakthrough was not adding another proxy. It was subtraction.

I had to remove cloud Grok as an available model path and force the shell onto local models only. No cloud default. No cloud web-search model. No fork secondary pointing at Grok. Just the local stack.

Only then did the setup behave the way I expected.

That was the second practical lesson:

Partial local is not local.

If the agent can still choose a cloud model, fall back to a cloud model, or require cloud identity because a cloud model remains in the catalog, then you are still operating a hybrid system.

Hybrid may be fine. But it should not be mistaken for local.

What Happened When It Worked

The local stack ran, but small models were not reliable enough for real multi-repo work.

Once the plumbing was up, I ran real agent work against the local setup.

This is where the romance wore off.

The local models were interesting. They were not useless. But they were not close to Claude Code or cloud Grok for the kind of multi-repo engineering work I actually do.

On my laptop, 8B-class models were slow. Not “a little slower.” Sometimes painfully slow. A cold model or a stressed machine could turn a simple response into a long wait.

Worse, the agent often sounded more competent than it was.

It would tell me it created files.

Then I would check the disk, and the files were not there.

It would say it fixed a config.

Then I would open the config and find invented keys, mixed formats, or a TOML file with YAML-shaped content pasted into it.

One specific test made the problem obvious. I asked it to create three Grok config profiles:

• Stock Grok Cloud
• LiteLLM orchestrator Grok
• no-auth local Grok

The local agent had the working config on disk. The right answer was mostly to inspect it, copy the good structure, and make careful variants.

Instead, it fabricated structure.

That is not a small problem for a coding agent.

When a chatbot hallucinates, you can usually see it in the text. When a coding agent hallucinates, it may claim to have changed the filesystem, updated the config, or completed the task.

The disk is the audit log.

In this case, the disk said the agent had not done what it claimed.

That was the third practical lesson:

For coding agents, honesty about tool results matters as much as raw intelligence.

A model that writes fluent explanations but cannot reliably use tools is not ready to own engineering work.

What The Local Trial Proved

Local-first is promising, but today it is still a lab path for me, not the production path.

The local-first experiment was still worth doing.

It proved that Grok Build can be used as a serious front end against custom model paths. It proved that local models can participate in the workflow. It proved that memory and rules can be moved across agent runtimes.

But it also proved the limits.

Here is where I landed:

1. A serious agent harness plus local open weights is real. It is useful for experiments, privacy testing, offline demos, and low-risk tasks.
2. Small local models are not interchangeable with frontier coding agents. The hard part is not writing plausible text. The hard part is reliable tool use, multi-file edits, and not lying about what happened.
3. Routers need to be tested at the tool-call level. LiteLLM may be fine for many workflows, but an agent harness can fail in ways that normal chat testing will not reveal.
4. Memory helps, but it does not fix weak execution. Rules and context help the agent behave better. They do not magically make a weak model reliable.
5. The “free local Claude-quality coding agent” story did not survive contact with my actual work.

I still believe in the direction. I do not believe laptop 8B models are the vehicle for my day-job coding yet.

That last point matters.

I am not against local models. I want local models to win.

But wanting something to be true does not make it production-ready.

Then I Looked At Kimi

Bigger open weights solve some quality problems, but private hosting creates a different problem: cost.

The next obvious question was:

If 8B local models are too weak, what about a stronger open-weight model?

That led me to Kimi.

Kimi is interesting because it is not a toy model. The Kimi K2 family is built for agentic work, and Moonshot’s own documentation describes serious tool-use and deployment paths.

But there is a big difference between:

I can call a hosted Kimi endpoint.

and:

I can privately host Kimi-class weights for my own coding agent.

That difference is the bill.

The convenient Ollama Kimi tags I saw were not giant private weights quietly running on my laptop. They were cloud-backed paths. That failed the privacy goal of the experiment.

True private Kimi-class serving is a multi-GPU problem.

Moonshot’s deployment guide for Kimi-K2 Instruct describes a smallest FP8 deployment unit around 16 H200/H20-class GPUs for the 128k context path. The Kimi-K2 Thinking guide describes an INT4 path with 8 H200-class GPUs.

That is not a hobby server.

Using AWS public GPU pricing as rough planning math, an 8-H200 shape is roughly a $50–$55/hour class resource, depending on the exact instance, region, and purchasing model. A 16-GPU deployment doubles that.

The math gets serious very quickly:

Those are not exact quotes. They are order-of-magnitude numbers.

But order of magnitude is all I needed.

For a small shop, “just host frontier-class open weights privately” is not a simple IP strategy. It is a serious infrastructure decision with real monthly burn.

That was the fourth practical lesson:

Open weights do not automatically mean cheap private inference.

The license can be open while the deployment economics are still very closed to a small operator.

Where I Landed: Grok Build With ZDR

ZDR is imperfect, but it is the best usable privacy control I can turn on today.

After the local trial and the Kimi cost check, my answer changed.

I no longer think the practical answer is:

Never use a cloud coding agent.

My answer is closer to this:

Use the best coding agent I can afford, but make the retention path explicit and verify it.

For me, that currently points to Grok Build using a Zero Data Retention path on my xAI API team.

xAI’s security documentation says ZDR is team-wide where available, that prompts and outputs are not persisted to disk for ZDR-enabled teams, and that features requiring server-side storage are disabled under ZDR. The docs also say ZDR can be enabled by a team admin in Console when available.

That is the right shape for my coding-agent use case.

I do not need Files, Collections, or server-side chat state for this path. I need an agent that can work against local repositories through a request path where prompts, outputs, and source-derived context are not persisted on the provider side.

There is an important distinction here:

ZDR is not a silver bullet. It is a property of a specific request path.

If I am doing sensitive coding work, I do not want to assume that a web subscription, a browser login, and an API team setting all mean the same thing. I want the coding path tied to the team/API configuration where ZDR is active, and I want to verify it where the platform provides a way to do so.

That is also why local configuration hygiene still matters:

• telemetry off
• feedback off
• no accidental cloud fallback models
• explicit model configuration
• loopback-only endpoints for local experiments
• clear separation between local lab work and cloud-agent work

ZDR does not make the cloud disappear.

Prompts still go to the provider for inference. Packets still leave the laptop. This is not an air gap.

ZDR is about retention posture. If the requirement is true air gap, the answer is local or private VPC inference, and then the cost and quality tradeoffs come back immediately.

I am clear-eyed about that.

Chamath Palihapitiya put it well on the All-In Podcast when he said, “privacy in AI is very fragile and it’s very brittle.” He went on to make the larger point that AI has all kinds of non-obvious data leak vectors, and that simply flipping a ZDR switch does not guarantee everything is safe.

I agree with that warning.

ZDR is not a silver bullet. It is the best control I can actually enable right now without removing AI from my workflow or renting an expensive GPU cluster.

And that is where Grok currently has a real advantage for me.

I trust Elon Musk and xAI enough to use the switch they actually gave me. Anthropic and OpenAI may have enterprise paths for larger customers, but as a small operator they have not given me a self-serve ZDR switch I can turn on for this workflow as xAI did.

It may not be perfect, but it is the strongest assurance I can actually buy today.

For my current situation, Grok Build plus a verified ZDR path is the most practical balance I have found:

• usable frontier coding-agent quality
• no private GPU cluster
• stronger retention posture than ordinary cloud use
• local models still available for experiments
• portable memory and rules I own

It is not perfect.

It is workable.

That matters more.

My Current Rule

The practical answer is not local or cloud. It is choosing the right control for the right work.

Here is where I landed:

Local-first when the work can tolerate the model quality, which is right now never, but maybe tomorrow.

Cloud agents when the work needs frontier-level coding ability, but only through a retention path I can verify.

No pretending those are the same thing.

Local models are part of my lab now, mostly so I can monitor their progress. They are useful. They are getting better. I will keep testing them. Maybe someday local models will be usable enough to give me an excuse to buy a couple of fully loaded Mac minis to host them.

But for serious multi-repo coding work, I still need a stronger agent.

The difference is that I am no longer treating the agent as magic.

I am treating it as infrastructure.

That means retention, identity, configuration, fallback behavior, tool execution, audit logs, and cost all matter.

In other words, the coding agent is not just another app on my laptop.

It is becoming part of the engineering process.

And anything that becomes part of the engineering process needs an architecture I can live with.

Sources

• All-In Podcast, July 18, 2026 — segment at 42:49: “Grok Build data leak, AI data privacy, Tokenmaxxing update, Mira Murati’s new model” — iHeart episode page and YouTube timestamp
• xAI Grok Build docs — Grok Build overview and custom models
• xAI security docs — Zero Data Retention FAQ
• xAI enterprise deployment docs — Grok Build authentication and network requirements
• Moonshot AI — Kimi-K2 deployment guide
• Moonshot AI — Kimi-K2 Thinking deployment guide
• AWS — EC2 Capacity Blocks for ML pricing
• Diginomica, July 3, 2026 — Alex Karp / CNBC tokenomics discussion — article
• CyberInsider, July 16, 2026 — Grok Build retention coverage — article
• SC Media, July 16, 2026 — Grok Build retention coverage — article

— Hector L. Alicea is the Principal of HLA & Associates, LLC, a federal technology consultancy specializing in platform engineering, enterprise integration, and AI-assisted software engineering. After nearly four decades building systems, he’s now exploring how AI changes the economics of software engineering.