I Spent Days Deploying a 3D Reconstruction Pipeline on Modal. Here’s Why I Went Back to RunPod.

I build AI/ML pipelines for a living. When I heard about Modal, thought to give it a try for our 3D reconstruction workload. I have thoughts. Let me save you the pain.

What I Was Deploying

A heavy ML pipeline considering COLMAP for camera pose estimation, GroundingDINO + SAM2 for masking, tiny-cuda-nn and nvdiffrast for neural rendering, pytorch3d for mesh processing. Its a full 30 to 45 minute GPU job per request.

This matters because Modal’s strengths and weaknesses show up very differently depending on your workload type.

Issues Faced During Image Build

Modal builds container images differently from Docker. It chains methods like .apt_install(), .pip_install(), .run_commands() and snapshots each one as a separate layer. Sounds clean.

Layer ordering is strict and non obvious. If you call add_local_dir() to copy source files, nothing can come after it, no pip_install, no run_commands. Files needed during the build (like a requirements.txt or a source package) need a copy=True flag. Everything else goes at the very end. The error messages when you get this wrong don't tell you why.

from_dockerfile has a symlink bug. If your build context has any system symlinks (like /run/udev/watch/ from your host machine's device manager), Modal's file scanner trips over them and crashes. Docker's build context never has this problem. Due to which deploying the Modal app from existing docker file was not possible and I had to rewrite the docker image content in python according to Modal’s instructions

The Python Version Trap

This one really got me. My Dockerfile installed everything under Python 3.10, torch, GroundingDINO, SAM2. Modal silently injected Python 3.11 on top and made it the default. My packages were all under 3.10. Modal’s runtime was using 3.11. Everything was installed, nothing was importable.

The fix is add_python=None in from_dockerfile() and you only find it after hours of confusion. And I couldn't just switch to 3.11 because SAM2 has compatibility issues with it. So I was stuck fighting Modal's runtime assumptions against my stack's requirements.

The 150 Second Wall

Modal web endpoints have a hard 150 second HTTP timeout. No matter what timeout you set on the function itself, the HTTP layer cuts you off at 2.5 minutes.

My jobs run for 30–45 minutes. So every single test was getting silently killed mid training, and I was getting back nothing. No clear error, just a dead request.

The fix I did was a spawn pattern where the endpoint returns immediately and the real work runs in the background, firing a webhook on completion:

@app.function(gpu="A100", timeout=60*90)
def run_job(event):
    handler(event)  # already calls a webhook on finish

@app.function()
@modal.fastapi_endpoint(method="POST")
def trigger(event):
    run_job.spawn(event)
    return {"status": "accepted"}

This works. But it’s a non obvious workaround for what should be a straightforward “run a long job” use case.

The Performance Problem

After sorting all of the above, I ran actual benchmarks. This is where I made my decision.

On RunPod with a consumer RTX 4090 (24GB): 10,000 training iterations in ~45 minutes.

On Modal with an A100 (40GB): 1,000 iterations in ~30 minutes.

That’s roughly 13x slower per iteration on paper but in wall clock terms for a full job, it meant Modal would take hours to do what RunPod does in under an hour. The culprit is almost certainly tiny-cuda-nn, which is heavily optimized for consumer Ampere/Ada architecture (the 3090, 4090, 4000 Ada). Data center A100s just aren’t its sweet spot.

Moreover another reason for me to not continue with Modal was the pricing difference, where Runpod (RTX 4090 24gb GPU) cost us 0.69$/hr Modal (A10G 24GB GPU) cost 1.10$/hr

Modal doesn’t offer commercial GPUs. That’s a deliberate product choice. For short inference workloads it doesn’t matter, instead it worked great. For my specific workload, it was a dealbreaker.

So When Does Modal Actually Make Sense?

Modal is genuinely well built for the right use case.

• Short inference jobs: the 1–2 second cold starts and pay per execution model shine here
• Spiky, unpredictable traffic: you’re not paying for idle containers at all
• Simple stacks: the image build quirks hurt a lot less when you’re not wrestling with 6 CUDA compiled packages
• Teams who want zero infra: no Kubernetes, no container registries to manage, just Python

But for long GPU training jobs, consumer GPU architecture requirements, or complex multi framework CUDA stacks RunPod serverless gives you more control, better hardware fit, and in my case, dramatically better performance for the workload.

TLDR

Modal has real promise and I get why people love it for inference APIs. But if you’re running heavy, long running GPU jobs with complex CUDA stacks, expect to fight the image build system, the Python runtime injection, the HTTP timeout wall, and potentially GPU architecture mismatch.