NVSentinel: Running your GPUs 24x7…

It has been long since I wrote my experiences here, partly have been busy with customers in helping them achieve their Gen AI goals as much as I can. That being said, I met a customer, a neo cloud, GPUaaS provider last week and one of the things they mentioned to me, was they were having tough time maintaining existing fleets of H200 SXM GPUs. This provider has 4000+ H200 all connected with state of the art IB backend in proper RAIL topology. Provider has left no stone un-turned to optimize the compute, network, storage etc. to ensure the end users have best outcomes they can for a fair price. However one of the key challenges, this provider often faces is GPU FATIGUE… Yes, even GPUs do get tired! When they do, the outcome is less than ideal:

• Silent data corruption (faulty GPUs producing wrong results).
• Cascading job failures (one bad node kills multi-day training runs).
• Idle healthy GPUs while waiting for manual triage.
• Pager fatigue for on-call teams (hours/days of manual diagnosis).
• Low utilization & SLA risk in shared/multi-tenant environments.

Provider is looking for a solution which just does not report on GPU setup and current utilization (DCGM) but also helps in remediating the issue and automatically removing the challenged GPUs from the production. This is where NVSentinel comes into play, it is NVIDIA’s open-source, Kubernetes-native fault detection, classification, and automated remediation system purpose-built for large-scale GPU-accelerated clusters. It continuously monitors GPU nodes (and NVSwitch fabric) for hardware and software faults, then automatically quarantines unhealthy nodes, drains workloads, and triggers repair workflows — without human intervention.

Key features include:

• Real-time, multi-source monitoring (GPU via DCGM, system logs, cloud provider events, custom Kubernetes objects).
• Policy-driven fault classification using CEL (Common Expression Language) rules.
• Automated node lifecycle: cordon → drain (with per-namespace strategies) → remediation CRD → optional reboot/terminate.
• Modular & pluggable architecture (use only the pieces you need: monitoring only, full closed-loop, or custom monitors).
• High availability (replicas + leader election, MongoDB change-stream driven).
• Metadata enrichment (cloud provider context, GPU topology, DCGM/driver versions).
• Event export in CloudEvents format for external systems.
• Graceful eviction with configurable timeouts and Slony drain support.
• Persistent event store (MongoDB) for auditing and pattern analysis.
• Kubernetes-native (NodeConditions, labels, CRDs for maintenance).

Deep Dive Architecture:

NVSentinel uses a disaggregated microservices design with independent modules coordinated via MongoDB change streams + Kubernetes API. All health monitors speak gRPC to a central Platform Connector.

1. Health Monitors (pluggable, extensible):

• GPU Health Monitor → DCGM diagnostics (thermal, ECC correctable/uncorrectable, XID errors, etc.).
• Syslog Health Monitor → journalctl for driver crashes, kernel faults, hardware errors.
• CSP Health Monitor → AWS/GCP/OCI maintenance & hardware events.
• Kubernetes Object Monitor → CEL-based policies on any K8s resource.

2. Core Processing Modules:

• Platform Connectors — gRPC server, validation, persistence, NodeCondition updates.
• Fault Quarantine — CEL rules → cordon node.
• Node Drainer — graceful pod eviction with namespace-specific strategies.
• Fault Remediation — creates Maintenance CRDs to hand off to external repair systems.
• Janitor — cloud-provider reboots/terminations.
• Health Events Analyzer — pattern detection & recommended actions.
• Labeler + Metadata Collector — auto-labels nodes, gathers GPU/NVSwitch topology.
• Log Collector — bundles diagnostics for troubleshooting.
• Event Exporter — streams to external SIEM/observability.

3. Data Flow (for the errors encountered, issues recorded):

• Monitors detect → gRPC event.
• Platform Connector persists to MongoDB + updates K8s.
• Watchers (Quarantine/Drainer/etc.) react via change streams.
• Actions executed on Kubernetes API + external hooks.

If NVSentinel detects an error or an issue, from one of its connectors (Continuous polling + event-driven (DCGM, logs, CSP APIs)), it goes through below process to make sure the administrator is aware of the issue and the unhealthy node is cordoned or removed from the fleet.

https://developer-blogs.nvidia.com/wp-content/uploads/2025/12/nvsentinel-remediation-workflow-png.webp

Supported Environments and Compatibility:

• Kubernetes: 1.25+ (tested on bare-metal, cloud, DGX Cloud).
• GPUs: All DCGM-supported data-center GPUs (H100, A100, B200, GB200/GB300, A30/A40, older Tesla, etc.).
• Platforms: DGX A100/H100, HGX A100/H100/B200, NVSwitch systems.
• Required: NVIDIA GPU Operator + DCGM (included).
• Optional: cert-manager, Prometheus, cloud-provider credentials (AWS/GCP/OCI), external repair systems.
• Integrations: Rafay (via Cluster Blueprints for fleet-wide consistency), NVIDIA GPU Health Service (fleet-level telemetry complement).

So for this GPUaaS provider we went ahead and have deployed this solution, allowing them to monitor and manage GPU failures and issues using NVSentinel. While this sounds pretty slick however as far as I know this tool has no support for SLURM driven fleets (correct me in comments, if I missed this), also this does not work with AMD GPUs (like obviously!). We expect issue detetcion and isolation to be reduced from Minutes vs. hours hence ensuring higher utilization and better SLAs for GPU clouds.

It seems like this has proven itself at scale since NVIDIA runs it on >40k GPUs internally; 13+ releases and external contributions by early 2026, more details here. If you have large NVIDIA GPU fleets, this tool is worth a try, from personal experience I can say, pretty staight forward to deploy and manage as well, if you are GPU provider, K8s is not an issue for you, and this tool integrates very well ! If you like my work, feel free to get me a cappuccino!