From Kernel Scheduler to Python Source Line: Tracing a GPU Stall End to End

A GPU that reports 97% utilization can still be the slowest part of a training step, and the reason usually lives outside the GPU: a CPU…