Two DGX Sparks Under My Desk: Building a Time-Series Monitoring Layer with InfluxDB

How I added persistent metrics, a 7-day history, and the foundation for an auto-remediation layer to a small Kubernetes cluster running on NVIDIA’s GB10-based DGX Spark workstations.

https://theonequote.app/quotes-about-growth/perfection-is-static-and-i-am-in-full-progress-anais-nin

Perfection is static, and I am in full progress. — Anaïs Nin

That line is a good description of infrastructure triage. You almost never get the perfect design on the first pass. You start with the shape of a problem, ship a small layer, watch where it breaks, and let the system teach you what the next layer needs to become.

That is exactly what happened here.

A live dashboard was already working. Then it became obvious that “live” was not enough. InfluxDB worked. Then the first deployment quietly proved my Ingress assumption wrong. The chart came up only after the cluster’s actual access pattern — MetalLB LoadBalancer IPs, not DNS — became visible.

The monitoring layer did not arrive fully formed.

It progressed.

Why: a live view isn’t enough

I have two NVIDIA DGX Spark workstations sitting on a shelf next to my desk — spark-2959 as the control-plane node and spark-ba63 as the worker. Each carries a GB10 superchip, 121 GB of unified memory, and a small Kubernetes cluster that hosts whatever I’m experimenting with that week: vLLM serving Qwen-3, ComfyUI, OpenWebUI, and a few cron jobs that scrape data overnight.

I had already built a small Flask UI — cluster-control-ui on port 8085 — that streams live host metrics over Server-Sent Events: CPU, memory, disk, network, and GPU utilization/temperature, refreshed every second.

It’s beautiful for the now.

It’s useless for any of these questions:

Did the GPU spike to 90 °C overnight while I was asleep?

Was that pod restart correlated with the memory climb on Tuesday?

Which jobs are quietly hogging the GPU when nobody’s watching?

The live view shows the present moment and forgets it.

I needed a memory.

And once I had a memory, I wanted something more ambitious: an action layer that could see “GPU temperature has been above 85 °C for two minutes” and gently scale a runaway deployment to zero before things get unhealthy.

This is the story of phase one — the persistence and visualization layer — and the small detours I hit along the way.

How: the decisions worth talking about

Why InfluxDB and not Prometheus

The default answer in any “I want metrics over time” conversation is:

Prometheus + Grafana + node-exporter + dcgm-exporter

It’s the right answer at scale.

For two nodes, it’s four extra services to babysit, and ARM64 builds have been historically uneven for the NVIDIA exporters.

I considered three options:

I picked the middle option.

InfluxDB 2.x has solid ARM64 builds, native retention policies — set “7 days” once and forget about it — and ships with a real query UI. That means I get ad-hoc exploration for free without standing up a separate Grafana.

The whole thing fits in a single Kubernetes pod.

Where to run it: on the cluster it monitors?

This is where I had to think carefully.

The natural answer was:

Deploy InfluxDB as a pod, give it a Longhorn PVC, done.

The uncomfortable counter-argument was:

If Kubernetes itself goes sideways, you’ve lost visibility exactly when you need it most.

I went with the cluster anyway, but with a mitigation baked in: the Flask app keeps the most recent ~30 minutes of metrics in an in-memory ring buffer and only writes to InfluxDB asynchronously.

If InfluxDB is unreachable, the live SSE view keeps working, the in-memory buffer keeps collecting, and the writer retries with backoff.

When InfluxDB comes back, the buffered points flush in order.

The eventual policy/action engine reads from the in-memory buffer, not InfluxDB — so actions don’t depend on the database being up.

In influx_client.py, this looks like:

class InfluxClient:
    def __init__(self, url, token, org, bucket):
        # ...
        self._ring: Deque[_PendingPoint] = deque(maxlen=RING_MAX)
        self._stop = threading.Event()

    def write_host_sample(self, sample):
        """Caller (Flask) is non-blocking: just enqueue."""
        # ... build points from the sample dict, append to ring

    def _run(self):
        """Background worker: drain ring, write batch, retry on failure."""
        while not self._stop.is_set():
            self._stop.wait(WRITE_INTERVAL_SECONDS)
            if not self._ensure_client():
                self._stop.wait(RETRY_BACKOFF_SECONDS)
                continue
            batch = self._drain_batch()
            try:
                self._write_api.write(bucket=self.bucket, record=batch)
            except Exception as exc:
                self._record_error(f"write: {exc}")
                self._requeue(batch)  # put points back, oldest-first
                self._stop.wait(RETRY_BACKOFF_SECONDS)

The whole class never raises into the caller.

The Flask request thread does one thing:

INFLUX.write_host_sample(sample)

And then it moves on.

All the failure handling lives in the background worker.

That design choice is the difference between “monitoring added another dependency” and “monitoring degrades safely when one layer is down.”

The detour: how do you reach a service from outside Kubernetes?

After deploying InfluxDB, I wired up an Ingress at influx.spark.lan and restarted the Flask app.

Then I watched it accumulate 511,087 dropped points while frantically retrying writes against a hostname that didn’t resolve.

last_error: "Failed to resolve 'influx.spark.lan'
             ([Errno -2] Name or service not known)"
consecutive_failures: 26398
points_dropped: 511087

I had assumed the cluster’s other apps were reached via Ingress + DNS.

They aren’t.

A quick kubectl get svc -A told the real story:

ollama              LoadBalancer   192.168.86.201:11434
openwebui           LoadBalancer   192.168.86.200:8080
theonequoteadmin    LoadBalancer   192.168.86.209:3000

Every other app uses MetalLB LoadBalancer services with fixed IPs in the 192.168.86.200–220 pool.

No DNS dependency at all.

Browsers and service-to-service traffic both use the LB IP directly.

The Ingress YAML I wrote was decorative. It would have worked for someone running Pi-hole or a local resolver, but it wasn’t the actual access pattern on this cluster.

The fix was a one-line change to the service:

spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.86.214   # next free IP in the MetalLB pool

…and an update to the install script so it waits for MetalLB to assign the IP, then writes that IP into the env file the Flask app reads.

Within seconds of restarting:

points_written: 96   (and climbing)
last_error: null
consecutive_failures: 0

The lesson, which feels obvious in hindsight: match the cluster’s existing access pattern. Don’t import a different one because it’s more “correct.”

Every other app on this cluster talks to itself via LB IPs. There was no reason for the new monitoring service to be the odd one out.

Bucket sizing and retention

InfluxDB’s DOCKER_INFLUXDB_INIT_RETENTION=168h sets a 7-day retention policy on the default bucket.

At 10-second resolution, with about 10 metrics per host across two hosts, that’s roughly:

10 sec/sample × 8640 samples/day × 7 days × 10 metrics × 2 hosts
≈ 1.2 million points/week

InfluxDB 2.x compresses time-series data aggressively — empirically this sits around 60–80 MB per week of raw points.

The 10 GB Longhorn PVC I allocated is overkill for phase one and gives me room for the per-process attribution data coming in phase two without thinking about it again.

The aggregateWindow Flux operator handles downsampling on the read side, so I don’t need pre-computed rollups for phase one.

The /api/metrics/range endpoint dynamically picks a bucket size based on the requested window:

if bucket_seconds is None:
    bucket_seconds = max(10, window_seconds // 360)

This caps any chart at ~360 points regardless of whether you’re looking at 15 minutes or 7 days, which keeps the chart responsive and the query cheap.

Whitelisting metrics in the API

Flux is a real query language, and /api/metrics/range takes user-controlled input.

I could have built a generic query proxy.

I chose to be boring instead:

_METRIC_WHITELIST = {
    ("host_cpu", "percent"),
    ("host_memory", "percent"),
    ("host_gpu", "util"),
    ("host_gpu", "memory_util"),
    ("host_gpu", "temperature"),
    ...
}

if (measurement, field) not in _METRIC_WHITELIST:
    return jsonify({"ok": False, "error": "..."}), 400

The UI only knows how to ask for the metrics in this set, so the whitelist is a no-op for legitimate use and a hard gate against anyone who tries to inject Flux through the query string.

Fewer things to think about during the next security review.

What we achieved

The Monitoring tab in the Flask UI now has a Historical Metrics section underneath the live cards.

A dropdown lets you pick:

Window: 15 min / 1 hour / 6 hours / 24 hours / 7 days

And another lets you pick:

Metric: CPU %, Memory %, GPU Util %, GPU Temp °C, GPU Memory %, Disk %

The chart shows both nodes overlaid, color-coded:

spark-2959 blue
spark-ba63 green

It also has adaptive auto-refresh:

• Once every 15 seconds for the 15-minute window
• Once every 5 minutes for the 7-day view

That way, we don’t hammer InfluxDB for views that barely change.

A button next to the chart opens the InfluxDB UI at the same LoadBalancer IP — http://192.168.86.214:8086 — for ad-hoc Flux queries, custom dashboards, and exploring the raw data.

Behind the scenes:

• A background thread in the Flask app polls both hosts every 10 seconds via the same SSH-based collector that powers the live view, and persists every sample to InfluxDB.
• An in-memory ring buffer holds 30 minutes of samples so transient InfluxDB outages don’t lose data.
• The systemd unit sources /etc/cluster-control-ui/influx.env with EnvironmentFile=-....

The leading - makes the env file optional, so the service still boots cleanly on machines where the monitoring layer isn’t installed.

Concrete numbers from the deployed system:

What’s next: from monitoring to action

This was deliberately phase one.

Phase two extends the metrics collector to capture per-process attribution:

• Which Linux PID is using GPU
• Which Kubernetes pod owns that PID
• Which namespace it belongs to
• Which pod is responsible for CPU/GPU/memory pressure

The pod mapping is resolved by parsing:

/proc/<pid>/cgroup

…and joining against a 30-second-cached:

kubectl get pods --all-namespaces -o json

The UI gets a Top Consumers panel that ranks pods by CPU/GPU/memory pressure.

Phase three is the action engine.

A YAML rules file might look like this:

- name: gpu-overheat-cooldown
  enabled: false              # explicit opt-in per rule
  when: "gpu.temperature > 85"
  for: 120s
  target: top_gpu_pod
  action: cooldown            # scale to 0, wait, scale back
  cooldown_duration: 300s
  reentry_lockout: 900s
  exclude_namespaces:
    - kube-system
    - gpu-operator
    - longhorn-system

A small evaluator thread reads from the in-memory ring buffer, checks dwell times and lockouts, and calls into the existing kubectl helpers in the Flask app.

Every action is logged to a separate InfluxDB measurement so you can see, six months from now:

This pod has been killed twenty-three times — maybe its memory limit is wrong.

The enabled: false default is the only safety gate.

There’s no dry-run-then-go-live workflow. Rules are off until you flip them on, and once on, they act for real.

That’s a deliberate tradeoff: fewer toggles to get confused about, at the cost of needing to be confident before flipping the switch.

Building this with Claude: the prompts that actually moved it forward

I should be honest: this entire monitoring layer — the InfluxDB manifests, the Flask integration, the install scripts, the UI panel — was built in a single evening of pair-programming with Claude, Anthropic’s coding agent running in my terminal.

About 700 lines of new code across nine files.

Without AI, this would have been a weekend project, easily.

What’s interesting isn’t that AI sped it up. That’s the obvious part.

What’s interesting is which prompts actually unlocked progress.

The prompts that mattered were almost always intent-level, not implementation-level.

The opening prompt

spark-2959.lan and spark-ba63.lan have the same DGX Spark configuration.
I need to design and add a constant monitoring layer. We already monitor
perf data via the Flask UI at cluster-control-ui. Add a layer of time
series db to store the data and an action layer to take actions, stop /
restart certain process / jobs if the cpu / gpu is too hot in
temperature and occupied for too long.

That’s the entire spec.

Notice what’s missing:

• No choice of TSDB
• No deployment shape
• No schema
• No rule language

What I gave Claude was vague high-level intent plus concrete pointers: the existing UI, the two host names, and the goal of cool-downs.

What I got back wasn’t code.

It was a comparison table:

Option A — Embedded, minimal (recommended for 2 nodes)
SQLite with a metrics_samples table, WAL mode, time-bucketed retention.
One sidecar thread in the existing Flask process polls every 10s and writes...

Option B — Standard stack
node_exporter + dcgm-exporter as systemd units on each node,
scraped by VictoriaMetrics single-binary...

Which path do you want — A (embedded sqlite + in-process policy)
or B (VictoriaMetrics + dcgm-exporter + Grafana)?

This was the first thing AI did that a junior engineer wouldn’t: it refused to start coding and forced me to make a real architectural choice.

Two minutes of reading saved a weekend of rework.

The pivot

I came back with:

shall we use InfluxDB to store last 7 days data? more towards option A
with an action layer to manage / pinpoint the process / job caused
the trouble to mitigate them or let them cool down for a bit time

Claude didn’t just accept the InfluxDB pivot.

It asked three questions back:

• InfluxDB host: spark-2959, or a separate VM? Disk budget ~2–3 GB for 7d at 10s?
• Default action posture: dry_run: true recommended, or live?
• Auth for actions: simple shared-token check now, or follow-up?

Each question forced a decision that would have bitten me if AI had just guessed.

When I answered “deploy to k8s on spark-ba63 worker, live actions are okay, leave auth for follow-up,” Claude flagged the second answer as risky and proposed a softer compromise: a per-rule enabled: false default.

That guardrail is now baked into the design.

This is where the Anaïs Nin quote fits the build most clearly.

Perfection would have meant trying to design the complete remediation framework before shipping metrics. Progress meant making one uncomfortable decision at a time, then adding a guardrail where the design was still soft.

When the build hit reality

After the deployment, the chart sat empty.

I sent a screenshot and wrote:

check why the deployed monitoring not showing stats: and also the way to
deploy influxDB, shall be allowed to be accessed like other app, such
as theonequoteadmin etc

What happened next is the part of AI-paired infra work that still feels slightly magical.

Claude ran:

kubectl -n monitoring get all
sudo cat /etc/cluster-control-ui/influx.env
curl -s http://localhost:8085/api/monitoring/status

It spotted:

Failed to resolve 'influx.spark.lan'

Then it ran:

kubectl get svc -A

It saw how the other apps were exposed, recognized the MetalLB LoadBalancer pattern, picked the next free IP from the pool, edited service.yaml, applied it, patched the env file, restarted the service, and verified with the API endpoint — all in one tool-using turn.

Twelve commands.

The same debugging loop, done by hand, would have taken me 45 minutes of context-switching between terminals.

That was the triage loop in miniature:

Assumption → failure signal → inspect reality → adjust design → verify

Not perfection.

Full progress.

What AI was good at, and what it wasn’t

The pattern I noticed across this build:

The prompts that moved things forward were never:

write me an InfluxDB service.yaml

They were:

shall we use InfluxDB

more towards option A

deploy to spark-ba63

check why monitoring isn't showing stats

Direction-setting and reality-checking, not specification.

The deeper shift

Building monitoring tooling used to mean: read three blog posts, pick a stack, install the agents, fight with the dashboards, write the rules, maintain it forever.

The bottleneck was almost never the typing.

It was the research, the integration with what already existed, and the dozen small decisions about deployment shape, naming, retention, and alerting.

What changed isn’t that AI writes the code.

It’s that AI collapses the research-and-decide loop.

I described what I had and what I wanted. AI came back with two costed options. I picked one. AI came back with three clarifying questions and a risk it had spotted. I answered. AI built it. When it broke, AI debugged it by reading the system itself instead of asking me to copy-paste outputs.

For a small team — or a single engineer with a homelab — this changes what’s worth building.

Custom monitoring used to mean:

I’ll just use Grafana. It’s not worth my time to build something tailored.

Now, something tailored to two specific machines, integrated with an existing custom UI, with a per-process action layer that knows about your namespaces and your safe-to-kill list, is an evening of pairing.

Not a sprint.

That’s the unlock.

Not faster typing.

Lower activation energy for building things that fit the shape of your actual problem instead of the shape of an off-the-shelf tool.

Reflections

Pick the boring access pattern. Half a million dropped points and a restart later, the lesson stuck: every other service on this cluster used LoadBalancer IPs, and the new one trying to be different is what broke. “Match what’s already there” beats “use what the documentation example showed” every time.

Make failure safe by default. The non-blocking writer + ring buffer fallback isn’t clever. It’s just admitting that the monitoring stack itself can fail, and that the live view shouldn’t be coupled to the historical one. Cost: ~50 lines of influx_client.py. Benefit: a Kubernetes hiccup degrades the system instead of breaking it.

Build the smallest end-to-end thing first, then iterate. Phase one is intentionally just persistence + visualization. No actions, no per-process attribution. This means the historical chart is shipping today, and I get a week of real data to inform what phase two and three actually need to look like — instead of designing them in a vacuum.

Carve out the security-relevant decisions early. The metric whitelist in /api/metrics/range took five minutes to add and made one entire class of injection attacks impossible. The auth question for the upcoming action endpoints is sitting in the backlog where I can think about it before those endpoints exist.

If you have a small homelab cluster and want a richer view than “the Kubernetes Dashboard plus htop,” InfluxDB 2.x as a single pod is a genuinely pleasant middle ground between SQLite-in-process and the full Prometheus stack.

The whole thing — manifests, install script, Flask integration, UI panel — is about 700 lines of new code.

Small, honest, and the GPUs are visibly cooler at night.

The monitoring layer is not perfect.

It is in full progress.

And that is exactly why it is useful.

AI only gets real when you stop talking about it and start building with it.Used well, it unlocks what wasn’t possible before — and as it evolves, it keeps opening new paths and redefining how we do the old ones. That’s what I share here — what works, what breaks, and what’s worth understanding more deeply. **Follow along and subscribe** if you want to stay close to the edge.

[embed]About — Doran Gao — Medium Read writing from Doran Gao on Medium. Doran Gao builds AI-powered products and systems. Creator of TheOneQuote.app…medium.com