My AI Designs Rockets in 6 Simulations. A ‘Dumb’ Baseline Nearly Kept Up.

Building the AI was maybe the easier part, I experimented where the simulations I ran are actually good is the part that humbled me, indeed it taught me why AI demos quietly lie.

Here’s my perception about “I built an AI that does so and so” posts, including the ones that I have come across and get impressed by, almost none of them checked whether if the AI was doing better or something stupid.

They run a demo, AI does it’s thing and everyone claps, nobody asks if this was actually replacing the simple approach or it could have done it better.

Hence I built this system of AI that designs rockets, then I forced myself to ask this boring question, now this is what happened.

What I built

I’m an aerospace engineer who has a bit of interest towards rocketry with a majors in data science, so a problem that is half flight dynamics and half machine learning is my idea of a good weekend.

The system does three things:

• Simulates 1,000 randomized rocket flights in RocketPy, a validated 6 degree of freedom flight simulator.
• Learns from them, a model predicts how high a rocket will fly from just its first couple of seconds, another flags weird flights.
• Designs. Given a mission, hit this altitude, stay under this hard safety ceiling, keep the rocket stable, leave the launch rail fast enough, it automatically figures out a rocket: which motor, how much nose weight, the launch angle, the parachute size.

The designing is the interesting part. And it runs on one strict rule.

The one rule: never trust the AI

AI models love to sound confident. Ask one to design a rocket and it will cheerfully tell you that the design is perfect, irrespective of good or bad, this for a chatbot, fine, but for something that’s supposed to keep a rocket under a legal altitude ceiling, that confidence is a liability, a hallucination.

So my designer is never allowed to believe itself.

Every proposal gets flown in the simulator. Only numbers the simulator actually produced are ever trusted.

The loop is simple: propose a change → simulate it → check it against the hard safety limits → correct → repeat. There’s a hard cap of 8 simulations so it can’t spin forever. If it finds a design the simulator confirms is safe and on target, it ships it. If it can’t, it says no-go, and tells you exactly which constraint beat it.

That’s the whole philosophy: propose freely, but verify against reality.

The part I’m most excited about: a team of arguing AIs

Here’s where it gets fun. Instead of one AI trying to balance everything at once, I designed the decision maker as a small team with deliberately conflicting jobs — like a real engineering review meeting.

• The Performance agent only cares about hitting the target altitude. It wants the biggest motor and the least dead weight. It does not care about safety, that’s someone else’s problem.
• The Safety agent is its natural enemy. It wants the rocket under the ceiling, stable, and fast enough off the rail. It pushes for smaller motors and more ballast-directly fighting Performance.
• The Recovery agent worries about where the thing lands. A bigger parachute means a softer landing but more drift in the wind; a smaller one drifts less but hits harder. It wants the rocket to come down somewhere you can actually walk to.

None of them can act. They just argue their corner. Above them sits an Orchestrator (the referee) who reads all three opinions plus the simulator’s verified numbers, weighs the trade-off, and picks exactly one change to try next. Or calls go / no-go.

The point of splitting it up this way is that the tension is visible and honest. Safety literally fights performance, out in the open, instead of getting blurred inside one model’s head. It’s constraint satisfaction as a negotiation.

(A quick honesty note, because it matters later: for the clean experiment below, I actually ran a simpler, deterministic version of the decision-maker, not the full arguing-agents version, so that I was measuring the value of the loop itself, not the mood of a language model on a given day. Whether the arguing agents beat the simple policy is the next experiment I want to run.)

The boring question, made concrete

I could have stopped here, posted a video of the loop converging on a pretty rocket, and collected the claps. Instead I set up a fair fight.

I generated 100 random missions (different target altitudes, ceilings, and winds) and ran three designers over the exact same set:

• One-shot — the “dumb” baseline. Make one smart guess, commit, done. No iterating. I made it as strong as I fairly could: it uses its single allowed simulation to extrapolate the right motor.
• The closed loop — my system. Propose, verify, correct, up to 8 sims.
• Exhaustive search — brute-force every motor-and-ballast combination with the simulator. This is the best anything could do, and it reveals which missions are even solvable in the first place.

Then I measured what actually matters: did the simulator confirm a safe, on-target design? How many simulations did it cost? And my favorite → how often did each method ship a broken design while claiming it was fine?

The results humbled me (in the best way)

I expected my loop to blow the dumb baseline out of the water on success rate.

It didn’t.

One-shot — the “dumb” baseline

Solved 89% of solvable missions Shipped broken designs 69% of the time Used just 1 simulation

The closed loop — my system

Solved 100% of solvable missions Shipped broken designs 0% of the time Used ~6.3 simulations

Exhaustive search — the theoretical ceiling

Solved 100% Shipped broken designs 0% Used 28 simulations

Three takeaways — two wins and one honest miss.

1. It hits the best-possible answer for a fifth of the cost. My loop solved every solvable mission, matching brute-force exhaustive search exactly — but used about 6 simulations instead of 28. Same optimal outcome, a fraction of the compute. That’s a win I can defend without flinching.

2. It never ships a broken design. The one-shot baseline, unable to check itself, committed to designs the simulator later proved invalid on 69% of missions. Mine: zero. When “invalid” can mean “breaks the altitude waiver,” that gap is the entire point.

3. But it did not solve meaningfully more missions than the dumb baseline. On solvable missions, one-shot already nailed 89%. The jump to my 100% wasn’t statistically significant. If I’d written “my AI solves more problems!” I’d have been lying with a straight face. It doesn’t. Its value is efficiency and reliability, not raw capability.

That third point stung a little. It’s also the most useful thing I learned.

The trap I almost walked into, this is the part I’m quietly proud of.

If you rank the designers by accuracy, how close they land to the target altitude — the dumb one-shot looks better. It lands closer on average, and the difference is statistically significant. A lazier write-up would’ve buried that number or spun it away.

But it’s a trap. The one-shot lands closer precisely because it ignores the safety limits. It aims straight at the altitude and fires — the same recklessness that makes it ship broken designs 69% of the time. My loop “gives up” some altitude accuracy on impossible missions specifically so it can respect the ceiling and correctly refuse.

Accuracy, reported by itself, rewards the unsafe method. You have to show it next to the violation rate or the number lies to you.

That felt like the most “real research” moment of the whole thing.

An unglamorous war story

Not all of it was tidy statistics. I also wanted to export the finished designs into OpenRocket, the tool actual hobbyist rocketeers use. That became a multi-hour debugging saga: reverse-engineering a file format from real examples, discovering the online docs described an unreleased version of the software, and finding a genuine bug where I’d positioned the motor by its nozzle instead of its front edge — so it poked out the back of the rocket like a misplaced tailpipe.

Real engineering: 10% clever architecture, 90% “why is the motor sticking out.”

What I actually took away

• A verification loop’s value isn’t always “solves more.” Sometimes it’s “reaches the best answer cheaply” and “never ships garbage.” Less flashy — but for anything safety-critical, far more important.
• Metrics lie when they’re alone. Accuracy without a violation rate actively cheered for the wrong method.
• Baselines are humbling, and worth every minute. Without the dumb floor and the brute-force ceiling, I’d have had a demo and a vibe. With them, I have a measurement.

The obvious next step: everything above used the simple deterministic policy, not the arguing-agents version. The genuinely open question is whether reasoning agents- negotiating performance against safety — can expand what’s solvable at all, not just do it more cheaply. That’s the experiment I want to run next.

Want to poke at it?

The whole thing — the simulator pipeline, the ML models, the design loop, the arguing agents, and the study that humbled me — is open source:

[github.com/aaditya4401-dot/launchloop]

(https://github.com/aaditya4401-dot/launchloop)

If you build ML systems: run the boring baseline. It’s the difference between a demo and a result.

Written by Aaditya Edupuganti — aerospace engineer turned data scientist, built rockets in the past and honest experiments independently. If this was useful, a clap or two helps other people find it, and I’d genuinely love to hear what you’d test next.