I Gave My AI Agent 11 Math Tools. It Used 3.

The expensive lesson about the difference between building capability and getting activation.

— -

I spent three days building an 11-tool symbolic math stack.

SymPy for algebra.

Z3 for constraint satisfaction.

Prolog for logic puzzles.

Auto-detection hooks to route problems before the model even saw them.

4,000 lines of wrappers. 800 lines of tests.

Then I pointed it at the workload it was built for.

Ten days of telemetry later:

Out of 11 tools, the model picked up 3.

The other 8 sat on disk.

The auto-solve hook? 0.04% success rate.

Two injections. Out of 5,128 attempts.

This is the story of how I learned that building a tool and getting an agent to use it are completely different problems.

— -

## What I Built

| Tool | What It Does |

| — -| — -|

| algebra | SymPy solve, factor, expand |

| number_theory | is_prime, gcd, totient |

| linear_algebra | determinants, eigenvalues |

| stats | binomial_pmf, normal_cdf |

| logic | satisfiability, CNF conversion |

| set_tool | power_set, cardinality |

| units | dimensional analysis |

| chemistry | balance equations, molar mass |

| z3 | SMT-LIB constraint solving |

| prolog | Horn-clause logic |

| solve | Dispatch wrapper for all of the above |

Eleven tools.

The hardest part wasn’t the math.

SymPy and Z3 do the actual work.

The hard part was parsing.

Models say “x squared minus four” when SymPy wants x**2–4.

They write “prime(p) and p < 100” when Z3 wants a declare-const block.

So I built two assist layers:

Smart-template extractor. Regex for common math forms. Synthesize a tool-call shell. Model just fills in arguments.

Synthetic-solve. Run Z3 directly on extracted formulas. Inject the answer before the model even tries.

The model sees: “You already called z3. Here’s the result.”

Both layers exist because the model doesn’t reliably reach for math tools on its own.

— -

## What Actually Happened

Eleven days of runs.

Here’s what the model called:

| Tool | Invocations |

| — -| — -|

| number_theory | 16 |

| algebra | 6 |

| linear_algebra | 6 |

| z3 | 0 |

| solve | 0 |

| prolog | 0 |

| logic | 0 |

| set_tool | 0 |

| stats | 0 |

| units | 0 |

| chemistry | 0 |

Three of eleven got used.

Eight stayed dormant.

— -

## Why The Auto-Solve Hook Failed

The hook was supposed to bridge the gap.

Model won’t call Z3? Fine — detect the math, call Z3 myself, inject the answer.

Telemetry from 5,128 events:

| Outcome | Count | % |

| — -| — -| — -|

| No extract | 5,078 | 99.0% |

| Formula not Z3-friendly | 21 | 0.4% |

| Z3 failed | 6 | 0.1% |

| Success | 2 | 0.04% |

Read that bottom row again.

Twice. Out of 5,128 attempts.

The breakdown:

99% of the time, the detector found no math.

User prompts were natural language. “What’s the boiling point under conditions Y” looks like math to a human.

It doesn’t look like a Z3-decidable formula to a regex.

When the detector found something, Z3 couldn’t handle it.

Model tried to use prime as an atomic predicate. Z3 doesn’t have is_prime.

Formula needed real arithmetic. We set up integer arithmetic.

Timeouts.

The hook was the most-engineered component of the stack.

It contributed two answers out of thousands of prompts.

— -

## What The Metric Actually Showed

Pass rate went from ~2.9% (no math stack) to ~5.3% (full stack + 40 commits of tuning).

2.4 percentage points.

Real. Above noise.

Mostly attributable to number_theory.

When the prompt was clearly a number-theory question, the model reached for is_prime or gcd and got exact answers instead of hand-rolling buggy modular arithmetic.

algebra.solve contributed on polynomial roots. ~4 of 6 invocations flipped a case to pass.

linear_algebra was used 6 times and contributed nothing. The cases were already passing — the model handles small matrices fine with plain Python.

Z3, Prolog, logic, sets, stats, units, chemistry: zero contribution.

— -

## Why The 8 Dormant Tools Didn’t Fire

Three reasons:

1. The model doesn’t reach for them.

Tools it saw in training (SymPy basics, matrix math) get used.

Tools it hasn’t (Z3’s SMT-LIB syntax, Prolog’s Horn clauses) get ignored.

Even when the system prompt advertises them.

Adding a tool to the registry isn’t the same as making the model use it.

2. The router patterns didn’t match real prompts.

I built regex against my mental model of what prompts look like.

Real prompts look different.

“Find the smallest n such that n³ mod 13 = 7” reads like math to me.

The regex expects n³ mod 13 = 7 and sees “n cubed mod 13 equals 7.”

Miss.

3. The tools were too narrow.

chemistry.balance_equation takes ”H2 + O2 -> H2O”.

Real chemistry questions ask about equilibrium constants — three steps before balancing is relevant.

The tool handles step 3. The model needs help with step 1.

— -

## What I’d Do Differently

The biggest mistake:

Building 11 tools before measuring which 2 or 3 would get 90% of the lift.

If I were starting over:

1. Build the eval first.

Run the bare model. Categorize failures: missing facts vs missing computation vs missing reasoning.

2. Build one tool.

The one that addresses the most failure cases. Get it firing. Measure lift.

3. Build the second tool only when the first is saturated.

“Saturated” = cases in its category now pass reliably.

4. Track per-tool case impact, not just invocation counts.

A tool invoked 100 times with no case flips is dead weight.

A tool invoked 6 times that flipped 4 cases is gold.

I did it backwards.

I built 11 tools because they were interesting to build.

Three earned their keep.

The other eight are sunk cost.

— -

## The Lesson That Stings

Building capability and getting activation are different problems.

I had the capability.

Z3 can solve constraint problems.

Prolog can deduce logic puzzles.

The chemistry tool balances equations perfectly.

But capability sitting on disk is worth zero.

Activation is what matters.

Does the model reach for it?

Does the detection layer find the right prompts?

Does the tool shape match the problem shape?

The dormant tools aren’t broken.

They work in isolation.

They just don’t get activated by the workload that was supposed to feed them.

— -

## What Stays In The Codebase

number_theory.py, algebra.py, linear_algebra.py: earned their place.

The rest: dormant.

Kept because they cost nothing to keep.

The auto-solve hook: kept, with expectations reset.

It’s a 0.04% success rate component.

That’s not nothing.

But it’s not the bridge I thought it was.

— -

The headline result is that shipping more tools didn’t multiply the agent’s strength.

It moved a metric by 2.4 points and taught me that activation is a different problem from capability.

If I’d known that two weeks earlier, I would have built a much smaller stack and finished a much larger benchmark instead.

Hope isn’t a metric.

AI Disclosure: To enhance the writing process and streamline technical explanations, generative AI was utilized to assist with drafting, code optimization, or formatting parts of this article. All content has been reviewed, and refined by the author to ensure accuracy and quality.