Weekly AI Paper Notes — Measuring Agents in Production

I’m starting a new series to share the notes about the papers I have been reading recently. These papers are filtered out from thousands of Arxiv papers per week by a small personal project I built https://github.com/RedrumSherlock/arxiv-agent. Each week I set a personal goal to read at least one paper related to LLM inference, infrastructure, GenAI platform, agent memory, evaluation, etc., and I will share the notes I took from reading these papers here.

Overview

This paper Measuring Agents in Production is published on Dec 2, 2025, by UC Berkley Sky Lab, collaborated with Stanford, UIUC, and companies like Intesa Sanpaolo and IBM. If you don’t know the Sky Lab(and its predecessor AMP Lab), it is actually one of the top incubator of Data & AI infrastructure and platforms, where many influential systems/frameworks like Apache Spark, Apache Mesos, Databricks, vLLM, SkyRL, Ray, Alluxio originated from this lab.

This is an incredibly easy to follow paper, without any math or formulas or architecture like others. This is not a general review paper either, instead, it collected 300+ online surveys and 20 in-depth interviews with companies who were actually implementing GenAI agents in produciton, and studies the industrial trend over 4 topics.

Overall, it is a must read for any industrial leaders and agent developers.

Conclusion

The 4 topics they asked and the conclusion for each of them from this survey:

1. Why companies build agents? There is only one major motivation: to automate tasks, save human cost, and improve efficiency.
2. How are agents built? Mostly with close sourced models, no fine-tuning or RL invovled, mostly as workflows instead of autonomous agents.
3. How are agents evaluated? Mostly human as the judget, or use LLM-as-a-judge to automate it.
4. What are the challenges? No consistency, not always reliable is the biggest concern. Security, privacy, and latency are managable.

Some other interesting facts found from the paper:

1. Finance & banks are the most active adoptors of agents (40% of total use cases), which I can confirm from my personal experience. Out of the AI summit/conferences I went to in North America, a big portion of speakers/attendees are from financial institutes.
2. Agents that can be measured by ROI are much easier to push, which is also true.
3. Half of them have short (<500 tokens) prompts, but there is a significant long tail (12%) exceeds 10,000 tokens in prompts
4. 68% of systems execute fewer than ten steps and 46.7% with less than 5 model calls before requiring human intervention.
5. 40% of agent builders do not use any framework, and for the other 60% the front runners are Langchain/LangGraph and CrewAI (I guess Google ADK was not released or adopted when the survey was conducted, so it could take over some market share now). LangGraph is still the most popular one. For more serious developers like the 20 interviewed ones, 85% of them built the agent purely in-house without any framework.
6. Even though there are many benchmarks for LLMs, there is not one widely accepted for agents, as they are all domain specific. I doubt there will ever be one.
7. Open source models are only considered when there is data privacy or cost control requirements, which is what I am seeing at many places too.

Final Rating

I would give this paper 75/100 which is a pretty high score from my standard, mostly because:

1. These are real use cases in production, not only acedemic research
2. The observations are matching what I have seen in the industrial area
3. The paper asked some key questions and gave actual conclusion
4. The paper is written in a way that is very easy to read
5. The authors are from a key lab that has many connections to industrial companies thus I trust their results from those interviews

Some problems:

1. The sample size is still not big enough. I don’t fully trust those online surveys, and if the authors can extend the size of interviews from 20 to 100 this paper would be much more valuable
2. The GenAI tech stack involves too quickly, so the results in this paper might be outdated in just a few months.