The Fan-Speed Crime: Optimizing an LPBF Physics Solver — from PyTorch to Triton (10x Speedup)

Before we talk about melting GPUs, we should probably talk about why I decided to build a custom physics solver from scratch.

Recently, I wanted to dive much deeper into the world of Physics-Informed Machine Learning, specifically PINNs, Neural Operators, and Physics-Regularized Surrogates. But to train these architectures, you need data. Not just any data, but dense, transient, physically grounded data. Given my professional background working with Additive Manufacturing and the simulation of Laser Powder Bed Fusion (LPBF) processes, I figured building my own thermal simulator would be the perfect sandbox to generate these datasets.

Let me establish a quick disclaimer: This simulator is not chasing absolute, industry-grade multiphysics perfection right now. With simplified homogeneous Neumann boundary conditions and a focus on pure heat conduction and phase transitions, it is currently designed to be a highly specialized, local data-generation engine for ML models. (Though I am absolutely eyeing domain decomposition and full multiphysics for the future).

Figure 1: What 67.1 million voxels look like. A high-fidelity multi-hatch simulation of SS316L steel at 1 µm resolution. Running a grid of this scale (1024 x 512 x 128) locally on an eGPU is exactly why standard PyTorch had to be replaced with custom Triton kernels to keep memory from overflowing.

But to generate enough data to train a neural surrogate, the simulator needs to be fast. And that is where things went off the rails.

TL;DR: If you are just here for the code, you can find the complete PyTorch/Triton implementation in the GitHub Repository here.

1. Introduction: A Simple Optimization Turned Hardware Survival

If you run 3D simulations for Additive Manufacturing (LPBF) locally on an eGPU, you will eventually learn to understand your hardware the hard way.

My project setup was modest: a local laptop paired with a 16GB VRAM eGPU running on Windows (via WSL2). I had successfully implemented a Finite-Differences Method solver using PyTorch to simulate transient heat conduction, phase changes, and non-linear material properties. Why PyTorch? Because Differential Physics is beautiful, and I wanted to keep the door open for backpropagation.

But as the simulation fidelity grew, my VRAM evaporated. The baseline PyTorch solver was eating 1.76 GB of memory for a moderately sized grid, and it was painfully slow. I decided to rewrite the core thermal stepping logic using OpenAI’s Triton.

The goal was a simple optimization. The result? A 10.6x speedup and a 70% reduction in peak VRAM (down to 0.53 GB).

But it came at a severe cost. I ended up fighting two “final bosses” of local hardware development: the Windows TDR Watchdog (which aggressively killed my drivers) and a thermal ceiling that drove my GPU above 95°C, forcing me to manually override the fan curves to a screaming 100% duty cycle.

Before we get to the hardware crashes, let’s look at why PyTorch failed, and how Triton’s black magic fixed it.

2. The Baseline: Why Local PyTorch Hits a Wall

When simulating heat conduction, we need to calculate the divergence of the conductive heat flux: Div(k * Grad(T)). In a 3D Finite-Difference scheme, this means looking at every single voxel, checking the temperatures of its 6 immediate neighbors, calculating the thermal conductivity at the interfaces, and updating the heat flux.

My initial PyTorch approach (ops.py) did this exactly how the deep learning textbooks teach you: heavily vectorized operations.

2.1. The Padding Penalty

To enforce Homogeneous Neumann Boundary Conditions (ensuring no heat magically escapes at the edges of the simulation domain), I used PyTorch’s padding function:

# Padding the entire 3D grid to enforce boundary conditions
pad = (1, 1, 1, 1, 1, 1)
T_p = F.pad(T, pad, mode="replicate")
k_p = F.pad(k, pad, mode="replicate")

Every time this runs, PyTorch reads the entire temperature field from the VRAM, adds a border to it, and allocates a brand-new, even larger tensor back into the VRAM.

2.2. The Intermediate Tensor Trap

Next, I calculated the thermal conductivity between voxels using a harmonic mean:

# Vectorized flux calculation for the X-axis
Tx_l = T_p[..., 1:-1, 1:-1, 0:-2]
kx_l = harmonic_mean(kc, k_p[..., 1:-1, 1:-1, 0:-2])
flux_x_l = kx_l * (Tc - Tx_l) / dx

Under the hood, every single one of these lines triggers a separate CUDA kernel. PyTorch reads two fields, computes the harmonic mean, and writes an Intermediate Tensor back to the VRAM. Then the flux calculation reads that intermediate tensor again.

2.3. The Memory-Bound Dilemma

This is why my VRAM was constantly maxed out, and why the simulation was slow. For a single micro-timestep, gigabytes of data were being shoveled back and forth across the GPU’s memory bus. The actual processing cores (the ALUs) were doing almost nothing. They were just sitting there, cooling down, waiting for the VRAM to deliver the next chunk of data.

The system was heavily Memory-Bound.

3. Enter Triton: The Paradigm Shift

To break this bottleneck, I rewrote the solver in Triton (triton_ops.py). The objective was to eliminate all intermediate VRAM allocations and force the GPU to do all the math for a 3D block in its ultra-fast, on-chip SRAM before ever talking to the main memory again.

Before we dive in further: Why Triton? While JAX or native C++ CUDA kernels are valid alternatives, Triton offered the best trade-off for my workflow. It provides a robust balance between a ‘pythonic’ development experience and the raw performance of hand-tuned CUDA, without the overhead of a full C++ toolchain. For my local development setup, this was the perfect way to maintain fast iteration cycles while gaining the granular control over kernel-fusion layouts that I needed.

3.1. The Triton Black Magic (or: ELI10 for Engineers)

If you look at the triton_ops.py code, your first thought might be: What kind of Python/C-pointer fever dream is this? I get it, let's break down why this implementation is so obscenely fast (and why it turned my laptop into a screaming space heater…). Let's do an ELI10, but for engineers.

To understand the 10x speedup, you have to think of the GPU as a massive manufacturing plant. In this factory, we have three main components:

1. The Workers (ALUs): They do the actual math. They are crazy fast, but they have zero memory.
2. The Main Warehouse (VRAM): Huge (16 GB on my eGPU), but it’s geographically in another zip code. Walking there takes ages.
3. The Workbench (SRAM / Registers): Right in front of the worker. It’s tiny, but grabbing a tool takes literal nanoseconds.

Figure 2: Architectural comparison of memory-access patterns. Left: Standard PyTorch operations force high-latency VRAM communication due to implicit, opaque caching. Right: Triton kernel fusion enables explicit, high-bandwidth SRAM usage, keeping the entire computation chain on-chip. (Diagram conceptualized by me, generated with AI support (NotebookLM)).

To be pedantic: While actual L1/L2 caches are managed by the hardware controller rather than the user, this architectural comparison highlights the difference in explicit memory control. Triton enforces a tiling-strategy that aligns the data flow with the GPU’s memory hierarchy, making the cache effectively predictable and efficient , unlike the implicit, opaque nature of standard PyTorch operations which frequently force data back to VRAM.

The PyTorch Way: The Inefficient Worker

In my old PyTorch setup (ops.py), the workers were terribly managed. For the first step “padding the grid to enforce Neumann boundary conditions” a worker ran to the warehouse, grabbed the temperature field, glued a border to it, and walked all the way back to store the new, even larger tensor in the warehouse.

Next step? Walk back, get the data, calculate the harmonic mean for the thermal conductivity, and walk back to store another intermediate tensor. The workers spent 90% of their time waiting for the forklift to bring them data. The system was severely Memory-Bound. That’s why a moderately sized 3D grid casually ate up 1.76 GB of VRAM.

The Triton Way: The Assembly Line Revolution

Triton fundamentally changes this by enforcing Kernel Fusion. We don’t let the workers wander around anymore. Instead, we put a box of work on their workbench and tell them: “Do everything at once, and don’t you dare call the OS before you are completely done.”

Here is how that actually looks in the code:

1. Claiming the Box (Tiling): The GPU has thousands of workers. To prevent them from tripping over each other, the 3D grid is chopped into tiny blocks (e.g., 8×8×4 voxels).

pid_x, pid_y, pid_z = tl.program_id(0), tl.program_id(1), tl.program_id(2)
o_x = pid_x * BLOCK_SIZE_X + tl.arange(0, BLOCK_SIZE_X)[:, None, None]

Each worker asks the system, “Which box is mine?” (tl.program_id) and calculates their local coordinates.

2. The Single Warehouse Trip (Loading): Now, the worker grabs the temperature data and the material mask for their specific block from the VRAM and puts it on their workbench.

Tc = tl.load(T_ptr + idx_c, mask=m_all)
mc = tl.load(mask_ptr + idx_c, mask=m_all).to(tl.float32)

That mask=m_all is your safety harness. If a worker is at the very edge of the simulation domain, this mask stops them from blindly reaching outside the factory walls and causing an immediate segfault.

3. Workbench Math (The Register Magic): From this point on, everything happens exclusively in the registers (the workbench). Are we hitting the Solidus temperature and melting the powder? Let’s check:

# Promote powder mask to consolidated (1.0) if we hit solidus
mc = tl.where(Tc >= T_sol, 1.0, mc)

Quick tangent: GPUs absolutely despise standard if/else statements because they stall the assembly line (branch divergence). Triton uses tl.where instead: just calculate both outcomes and pick the right one without breaking stride.

While at the workbench, the worker linearly interpolates the non-linear material properties from the Lookup Tables (LUTs) and calculates the thermal divergence for all three spatial axes. All those intermediate variables (k_xl, div_x, rhs)? They never touch the VRAM. They exist fleetingly in the silicon and vanish once they aren't needed. This is exactly where our 70% VRAM reduction comes from.

4. Shipping the Product (Storing): Only at the very end, once the brand-new temperature for the next sub-step is fully calculated, does the worker take the final product back to the warehouse:

tl.store(T_new_ptr + idx_c, Tc + (dt / rho) * rhs / (cp_total + 1e-9), mask=m_all)

The Result: A Compute-Bound Revolution

By fusing the kernels, we violently shifted the bottleneck from Memory-Bound to Compute-Bound. The ALUs hit 100% utilization without taking a single breath. Because the workers no longer spend 90% of their time waiting for memory I/O, they are constantly performing heavy float arithmetic for the LUTs and phase transitions.

The code became so efficient that the hardware started pushing its thermal limits — a sign that we had finally extracted every ounce of performance from the silicon. But as I would soon discover during the dataset generation, this level of raw computational intensity comes with its own set of “final bosses.” Sometimes, good code doesn’t just need optimization; it needs a thermal survival strategy.

3.2. The Workbench Choreography (And Its Unforgiving Limits)

Now we know that the real magic happens on the workbench (the GPU registers and SRAM). But why exactly does the code look the way it does? Why do we load first, calculate like maniacs, and only save at the very end?

The reality of the hardware is strictly non-negotiable: The workbench is incredibly tiny, and the GPU is absolutely unforgiving of clutter.

When writing Triton kernels, you are constantly fighting two invisible final bosses: Register Pressure (running out of desk space) and the JIT Compiler (the factory foreman who hates surprises).

Limit 1: Block Size and the Nightmare of Register Spilling

Why do we pack the work into boxes of exactly 8×8×4 voxels? Why not 32×32×32?

Every single voxel we calculate needs physical space on that workbench. We load the center temperature field Tc, the spatial neighbors (Txl, Txr, Tyu...), the phase masks, and we calculate intermediate values like the thermal conductivity kc. All of this sits in the GPU's registers.

If we make the block size too large, the absolute worst-case scenario happens: Register Spilling. The workbench gets full, the worker drops their tools (data), and the GPU has to silently and secretly swap those intermediate results back into the excruciatingly slow main warehouse (VRAM) just to free up space. If that happens, your entire Triton speedup evaporates instantly. The dark art of kernel writing is loading exactly enough data to keep the registers 99% full, without spilling a single byte.

Limit 2: The JIT Compiler’s Wrath (No Unknown Variables)

Unlike C++ where you compile your entire program beforehand, Triton compiles the GPU kernels Just-In-Time (JIT), literally milliseconds before the Python script executes them. Because the compiler (the foreman) has to organize the factory floor in real-time while the conveyor belt is already moving, he absolutely refuses to deal with unknown variables. He demands strict, compile-time-known parameters, or he simply refuses to build the factory.

The Choreography: A Strict Order of Operations

To navigate these two limits without crashing, our kernel follows an extremely strict, borderline paranoid schedule:

Step 1: Secure the Raw Materials (The tl.load Phase)

First, we load all the temperatures and masks for the block.

Tc = tl.load(T_ptr + idx_c, mask=m_all)
# ... followed by all neighbors (Txl, Txr, Tyu, ...)

Why here? Because we want to get all the agonizingly slow VRAM waiting out of the way immediately. We batch the latency.

Step 2: Clean Up Before We Work (Inline Phase-Change)

Right after loading, we do something that looks a bit sketchy:

mc = tl.where(Tc >= T_sol, 1.0, mc)

The second the temperature hits the Solidus threshold, we overwrite the mask (powder becomes consolidated). Why don’t we load this from a dedicated tracker array? Because we don’t have the space! We modify the variable mc directly in the register to avoid allocating extra memory. We compute the physical phase transition entirely in-flight.

Step 3: The Lookup Table Limit (The Compiler’s Wrath)

To solve the heat equation accurately, we need temperature-dependent material properties (conductivity, heat capacity). These live in a Lookup Table (LUT). And this is where we hit Triton’s hardest limitation:

for i in range(16):
    if i < n_lut - 1:
        # Interpolate values from the LUT ...

Why is there a hardcoded range(16)? Why not just range(n_lut)? Because the Triton compiler (the strict foreman) must know at compile time exactly how many times this loop will run so it can reserve the exact number of registers on the workbench. It aggressively despises dynamic loop lengths. If we wrote range(n_lut), the compiler wouldn't know how much space to allocate and would straight-up refuse to compile the code. So, we make a blood pact with the compiler: "Listen, the table has a maximum of 16 entries, reserve the space for that." (Which is exactly why my Python wrapper throws a hard error if you try to feed it a 17-point LUT).

Step 4: Aggressive Garbage Disposal (Divergence Calculation)

Finally, we calculate the fluxes (div_x, div_y, div_z). The moment div_x is calculated, the worker no longer needs the temperatures of the X-neighbors (Txl, Txr). That precious register space is instantly freed up and cannibalized for the Y-axis calculations. It’s a constant, ruthless recycling of the exact same tiny piece of silicon.

Only when the entirely new temperature for this micro-frame is fully baked (rhs), do we call the forklift and push the result back to the VRAM via tl.store().

3.3. Battle Scars in the Code: 2 Hidden Hardware Traps

When you transition from high-level PyTorch to writing low-level Triton kernels, you quickly learn that elegant code is not always working code. My current script is full of small, seemingly irrational design choices. If you read the code closely, you will stumble across a few oddities (besides the hardcoded range(16) limit).

Here is the honest truth about why the code looks the way it does, and why the “cleaner” alternatives resulted in a catastrophic system failure.

Trap 1: The Asynchronous GPU Death (No tl.full Allowed)

Right in the middle of the phase-transition logic, there is this panicked comment: # CRITICAL-1 fix: use scalar 1.0 instead of tl.full(mc.shape, 1.0, ...)

What happened? In PyTorch, if you want to overwrite a mask with ones, you’d use something elegant like torch.full_like(). The first instinct in Triton is to do the same: if the temperature crosses the melting point, fill the mask with ones using tl.full(mc.shape, 1.0).

The problem is mc.shape. In the center of our 3D domain, every block is exactly 8x8x4 voxels. But at the boundaries of the geometry, the blocks are dynamically truncated to fit the edges. The Triton JIT compiler demands strict, compile-time-known shapes for tl.full. If you force it into dynamic shapes at runtime, it doesn't throw a polite Python ValueError. It triggers an Illegal Memory Access deep in the silicon. The GPU crashes asynchronously, taking the Windows driver down with it, leaving you with absolutely zero stack trace.

The Fix: We pass a naked scalar: mc = tl.where(Tc >= T_sol, 1.0, mc). Triton is smart enough to automatically broadcast this single 1.0 across the entire block, regardless of its dynamic size at the edges. It looks primitive, but it is bulletproof.

Trap 2: The PCIe Tollbooth (The lut_tensors Bypass)

In the Python wrapper function run_thermal_step_3d_triton, there is a parameter called lut_tensors. It exists solely to bypass the worst bottleneck in local hardware development: the PCIe bus.

When I first ran the Triton solver, it was crawling. The reason? The simulation configuration (material properties) lived in the CPU’s memory space. For every single micro-timestep — and we do hundreds of them per frame — the script called torch.tensor(mat_cfg.T_lut). This forced the CPU to package the data from the RAM and send it through the narrow PCIe bus to the GPU. The blazingly fast GPU had to completely halt its assembly line to wait for the CPU's slow delivery.

The Fix: The Lookup Tables are now loaded once into the GPU’s VRAM at the very beginning of the simulation (passed as lut_tensors). The wrapper function only accepts these GPU-native references. The CPU just fires the starting gun, but all the data stays permanently in the fast VRAM until the simulation finishes.

Trap 3: The NaN Poisoning (Epsilon Engineering)

If you look closely at the math inside the Triton kernel, you will notice tiny numbers scattered throughout the denominators, like 1e-9 or 1e-12.

For example, when calculating the harmonic mean for thermal conductivity between two voxels, the equation looks like this: ((2.0 * kc * k_xr) / (kc + k_xr + 1e-12) * ...)

Why add this microscopic 1e-12? Because in an explicit physics solver simulating extreme temperature gradients, you are constantly dividing by values that might briefly approach zero (like identical conductivities or temperature deltas during phase changes).

In standard Python, dividing by zero gracefully throws a ZeroDivisionError. In PyTorch, it generates a NaN (Not a Number) tensor, which is annoying but easy to track down.

In Triton? A NaN in a fused kernel is highly radioactive. Because our GPU workers are passing values locally in the registers at lightspeed, a single NaN generated in one voxel instantly infects all its neighbors during the next divergence calculation. Within milliseconds, the entire 3D grid evaporates into a chaotic soup of NaNs, and the simulation explodes silently without the compiler ever raising a red flag.

Sprinkling these tiny “Epsilons” into the denominators is dirty, low-level epsilon engineering. But it acts as the cheapest, most effective insurance policy to prevent a single floating-point anomaly from tearing a hole in the space-time continuum of your physics engine.

3.4. The Blueprint: Rules of Engagement for Triton Kernels

So, what is the takeaway here? Simply put: you need a survival guide... Based on my crashes, debugging sessions, and thermal throttles, here is my personal blueprint for writing Triton kernels without nuking your local setup:

Rule 1: Treat the Compiler like a Paranoid Bureaucrat

The Triton JIT compiler hates surprises. If it doesn’t know exactly how much memory to allocate before the code runs, it will either refuse to compile or crash your GPU.

• The Rule: Hardcode everything you can. If you have a loop (like my LUT interpolation), give it an absolute maximum range (e.g., range(16)) instead of a dynamic variable.
• Masks: If you need to manipulate masks at the edges of your geometry, never try to dynamically match shapes. Broadcast naked scalars (like 1.0) and let Triton handle the fitting.

Rule 2: The st.local Kiss of Death (How to check your limits)

How do you actually know if your workbench (registers) is full and you are spilling data back into the agonizingly slow VRAM? You look at the machine code. Before running your massive simulation, set this environment variable in your terminal: export TRITON_PRINT_PTX=1. This forces Triton to print the raw assembly code (PTX) it generates for your Nvidia GPU.

• What to look for: Search the output for the commands st.local (store local) and ld.local (load local). "Local" here is Nvidia-speak for VRAM allocated specifically for spilled registers. If you see these commands, your kernel is spilling. Your block size (e.g., 8x8x4) is too large, or you are holding too many variables at once. Shrink the block until the .local commands vanish.

Rule 3: Enforce a Strict PCIe Quarantine

The CPU and the GPU should only talk at the very beginning and the very end of the simulation.

• The Rule: Never load configurations, LUTs, or tiny parameter tensors inside your time-stepping loop. Push absolutely everything into the GPU’s VRAM during initialization. Your Python loop should only contain the kernel launch and a torch.cuda.synchronize() call if you need to appease the Windows TDR watchdog.

Rule 4: Branching is Illegal

Standard if/else logic forces the GPU's assembly line to halt, evaluate, and potentially split workers into different paths (branch divergence). GPUs despise this.

• The Rule: Calculate everything, everywhere, all at once. Use tl.where(condition, true_result, false_result). You make the workers compute both the true and the false outcome, and just throw away the one you don't need. It sounds incredibly wasteful, but on a GPU, doing redundant math is significantly faster than stopping to think.

Armed with these four rules, my kernel was finally bulletproof. But as I was about to find out, a bulletproof software kernel doesn’t protect you from a hardware meltdown.

4. The Dataset Horrors (Or: A Fan Speed Crime)

With the Triton kernels heavily optimized and the VRAM footprint minimized, I finally had a blazing-fast solver. Now, I needed data.

I wrote a script randomizing the material LUTs, process parameters, and hatch patterns (how the lasers move across the powder bed) to generate a holistic dataset of ~2,000 samples. With full confidence, I started my script, leaned back, and waited for my sweet data.

Aaand… WHY IS MY GPU CRASHING?

2026-05-03 19:24:30 [INFO] --- Run 1 | Mat: SS316L (Randomized) ---
Simulating:  73%|███████▎  | 54/74 [06:34<02:26,  7.31s/it]
Traceback (most recent call last):
  File "scripts/generate_offline_dataset.py", line 252, in _run_worker
torch.AcceleratorError: CUDA error: unknown error
# Driver crashed in the background during Triton execution.
# Subsequent copy to CPU memory reveals the disconnected device.

I had encountered two distinct “final boss” bugs that I had to untangle. One software-based and one hardware-based. As it turned out, they were independent hurdles on the way to a stable dataset.

4.1. Horror #1: The Windows TDR Watchdog (The Software Wall)

In my Jupyter Notebook prototypes, everything ran perfectly. So what changed in the script?

I quickly learned something new about how Windows Subsystem for Linux (WSL2) handles hardware. Windows monitors GPU activity via the Timeout Detection and Recovery (TDR) watchdog. If a single GPU operation occupies the device for more than ~2 seconds, Windows assumes the GPU is frozen and forcibly resets the driver.

High-diffusivity materials trigger significantly more CFL (Courant-Friedrichs-Lewy) sub-steps per macro-step. During long laser exposure times (≥50 µs), individual step_adaptive calls were hitting this 2-second limit.

My first attempt to fix this was chunking the execution. Instead of calling stepper.step_adaptive once for the full exposure time, the script iterated in smaller chunks of at most 5 µs each. I restarted the dataset generation... and it crashed again.

The missing link: PyTorch kernel launches are asynchronous. The step_adaptive function returns control to Python the moment the kernels are queued, not when they finish. Without a sync point, my while-loop enqueued the next chunk's kernels before the GPU had finished the previous ones. To the Windows TDR watchdog, consecutive chunks appeared as one continuous, uninterrupted block of GPU work.

The fix was forcing a hard synchronization. It gives Windows the “breathing point” it needs to reset its 2-second TDR timer between slices.

# The Fix: Chunking execution and forcing a GPU sync
for chunk in time_chunks:
    stepper.step_adaptive(chunk)

    # Crucial for WSL2/Windows: Force PyTorch's async kernel queue to finish.
    # Gives the OS a "breathing point" to reset the 2-second TDR timer.
    torch.cuda.synchronize()

I was certain this was it. I fired up the script. NOPE. I even went into the Windows Registry to force the TDR watchdog to wait a full 60 seconds (TdrDelay) before sticking its nose into my simulation. AGAIN, A BIG NOPE.

4.2. Horror #2: The Thermal Ceiling (The Hardware Wall)

Why was my GPU still crashing? And this wasn’t a simple Out-of-Memory error. The GPU genuinely disconnected from the laptop. The driver suffered a full-blown hiccup.

Remembering that I am an engineer, I decided to stop guessing and start measuring. I implemented a comprehensive tracking system acting as my flight recorder, logging system metrics alongside the simulation outputs.

The answer to my problems was as simple as it was embarrassing: Triton worked too well.

While the VRAM was only 50% utilized, the GPU’s processing units were under extreme strain. Where standard PyTorch usually has I/O pauses between operations (waiting for memory transfers), Triton keeps the arithmetic logic units (ALUs) running at full throttle without a single pause.

Because we calculate nonlinear material properties for each grid point locally in the registers, we perform a massive number of mathematical operations per loaded data point. We violently shifted the bottleneck from Memory-Bound to Compute-Bound. The GPU cores had to work relentlessly instead of waiting for data.

The telemetry showed the grim reality: The GPU rapidly hit 95°C. To survive, the hardware throttled the base clock frequency down to a miserable 300 MHz (explaining the sudden drop in iteration speed), right before the driver pulled the plug entirely to prevent silicon damage.

The Solution: I committed the ultimate Fan Speed Crime. I manually overrode my laptop’s thermal management, cranked the fans up to a deafening 100% duty cycle, and removed the carpet from beneath my cooling pad.

With the fans screaming, the GPU stabilized at 88°C. Those 7°C made the entire difference between a hardware disconnect and a stable system. Consumer-grade hardware used for highly complex technical tasks is really something special.

5. Outro: Moving Forward

Do I still need the kernel-slicing and torch.cuda.synchronize() now that the fans are drowning out my thoughts? Heck yeah.

While the fans fixed the thermal hardware disconnect, the kernel-slicing is still mandatory to bypass the Windows TDR watchdog. They are two separate safety nets for two separate failure modes: one for the OS, one for the silicon.

With the system now stable at a controlled 88°C, I finally generated my offline corpus of 650 unique, high-fidelity simulations (merging rapid prototyping datasets with a final ‘Hero Run’).

Figure 3: Dataset Sample 125. Multi-hatch thermal simulation of SS316L showing surface temperature, solidification history, and the normalized heat source across a 1.0 x 0.5 x 0.125 mm domain. The 512x256x64 grid resolves transient thermal effects down to 1.96 µm per voxel.

I can finally stop debugging hardware and move on to the fun part: training and optimizing the physics-informed AI model. But that is a story for part two.

The complete source code for this project, including the custom Triton kernels and the data generation pipeline, is open-source. You can dive into the repository here: https://github.com/technojesusB/learning-based-lpbf-thermal-design.

— -

P.S. — I am looking for my next adventure!

If your team is building cutting-edge Physics-Informed AI or applied machine learning systems and you need an engineer who isn’t afraid to push hardware to its absolute limits — let’s connect. I am based in Berlin and open to local or 100% remote roles (or roles with a reasonable hybrid model). Feel free to reach out to me on LinkedIn!