🥡 Would You Like an LLM with That? — A Bento Cloud Drive-Thru

Part one of the “Building a Hypermodern GenAI App” series

The most epic grilled cheese sandwich in the most epic bento. Image generated by ChatGPT

The generative AI space sometimes feels like a buffet – dozens of dishes that all taste suspiciously similar. By day, I’m using LangChain and my team’s go-to stack for LLMs, vector databases, and observability. By night, I use LlamaIndex and tinker with whatever shiny new toy catches my eye.

But here’s the thing: the deeper you go into these stacks, the more everything blends together. Read enough cookbooks and you’ll swear they were written by the same chef. In the end, it’s the same sunny side egg plated differently — the logo on the apron doesn’t change the taste

Introducing Hypermodernity

Add “hyper” to a word and it usually sounds negative – like hyperactive (too active). But in chess, hypermodernism has a more subtle meaning.

Classical chess training teaches you to control the center squares early with pawn moves. The hypermodern school flips this on its head: you let your opponent occupy the center, then apply pressure on the center squares, turning these squares into attack targets.

That’s exactly the flavor of this article series: a hypermodern approach to building GenAI apps. We’ll cook with modern tools, but instead of following the usual recipes, we’ll season them differently – rethinking the playbook, and serving up something a little unexpected.

We’ll phase our development across three “acts.” This article focuses on Act One: deploying an open-sourced LLM on a close-sourced platform without spending a dime.

All codes are available in my GitHub repository.

Introducing Bento Cloud

In my previous article, I introduced vLLM’s novel KV caching approach that sets it apart from other methods to serve an LLM — enabling it to perform up to 3x faster than Ollama while handling more concurrent requests. vLLM’s performance is out of this world. It’s likely adopted in the technology stack of many LLM vendors, even those serving open-sourced LLMs.

In that article, I demonstrated how to deploy a Llama 3.2 3Bn LLM with vLLM using a NVIDIA T4 GPU on a free-tier Google Colab runtime. Although successful, the approach of using one server wouldn’t work in production settings for apps with high user traffic. The sheer number of requests will cause performance issues, leaving many users waiting for the LLM’s response.

That’s why deploying a vLLM model is no mean feat. To handle high user traffic, deploying vLLM in production usually requires:

• Kubernetes for orchestration and load balancing across replicas.
• Terraform to spin up and tear down servers (to be used as replicas) by code
• Ansible for configuration management of the servers spun up by Terraform

That’s where Bento Cloud comes in.

Launched by BentoML in 2024, Bento Cloud extends the BentoML SDK with all the bells and whistles needed to get into production faster. With BentoML, you can wrap your application as a RESTful API, and it handles the hard parts like provisioning and autoscaling across replicas. It’s flexible too: deploy locally, on your own cloud, or straight to Bento Cloud — which is what we’ll do in this article.

For vLLM deployments, BentoML drastically lowers the production barrier. That’s why we’re exploring it here.

Getting Started

Create a free Bento Cloud account — new users get $10 in free credits for GPU servers.

Just remember: check the $/hr rate of each GPU type, and shut down deployments when not in use to stretch your credits.

The BentoML team have also kindly provided an example repository with many different LLMs served on Bento Cloud using vLLM. We will be adapting the codes here to deploy a Qwen3:8bn reasoning LLM.

What is a Bento? 🥡

A bento is the atomic unit of “stuff” in BentoML — like pods to Kubernetes and containers to Docker. Minimally, a bento comprises of a service, and a requirements.txt. Ours has a bit more:

🛎️ The Service

The service (or the “app”) contains the source code to be hosted on Bento cloud.

The Input ➡️

We’ll start by defining an input schema with Pydantic’s BaseModel to enforce strict type checking.

This schema will pass configuration options into BentoML, and most importantly, it’s what we’ll use to serve our vLLM model via vllm serve.

# service.py

from __future__ import annotations

import logging, json, os, typing, collections.abc, contextlib, httpx
import pydantic, bentoml, fastapi
from starlette.responses import RedirectResponse

logger = logging.getLogger(__name__)

class BentoArgs(pydantic.BaseModel):
    tp: int = 1
    attn_backend: str = ‘FLASHINFER’
    skip_flashinfer: bool = False
    piecewise_cudagraph: bool = True
    reasoning_parser: str | None = None
    tool_parser: str | None = None
    max_model_len: int | None = None
    autotune: list[int] | None = None
    hf_system_prompt: str | None = None
    include_system_prompt: bool = True
    sharded: bool = False

    ##### Most Important Arguments ####
    name: str = ‘qwen3-8b’
    gpu_type: str = ‘nvidia-h100-80gb’
    model_id: str = ‘Qwen/Qwen3-8B’
    v1: bool = True
    hf_generation_config: dict[str, float | int] = pydantic.Field(
        default_factory=lambda: {‘repetition_penalty’: 1.0, ‘temperature’: 0.6, ‘top_p’: 0.9}
    )
    ####################################

    post: list[str] = pydantic.Field(default_factory=list)
    cli_args: list[str] = pydantic.Field(default_factory=list)
    envs: list[dict[str, str]] = pydantic.Field(default_factory=list)
    exclude: list[str] = pydantic.Field(default_factory=lambda: ['*.pth', '*.pt', 'original/**/*'])
    metadata: dict[str, typing.Any] = pydantic.Field(
        default_factory=lambda: {
          'description': 'Qwen3-8B',
          'provider': 'Qwen',
          'gpu_recommendation': 'an Nvidia GPU with at least 80GB VRAM (e.g about 1 H100 GPU).',
        }
    )

A few important points:

• v1=True tells Bento to use vLLM v1 — which only runs on GPUs with compute capability ≥ 8.0 (e.g., NVIDIA H100). Note: Tesla T4s have a compute capability of 7.5.
• gpu_type (e.g., “nvidia-h100–80gb”) tells Bento Cloud which GPU to spin up.
• model_id points to the Hugging Face model you want to serve (e.g., “Qwen/Qwen3–8B”).

The other arguments have safe defaults — you only need to touch them if you want deeper customization.

The App ✨

# service.py

bento_args = bentoml.use_arguments(BentoArgs)

image = (
    bentoml.images.Image(python_version='3.12')\
    .system_packages('curl', 'git')\
    .requirements_file('requirements.txt')
)
if POST := bento_args.post:
    for cmd in POST:
        image = image.run(cmd)
if not bento_args.skip_flashinfer:
    image = image.run(
        'uv pip install --no-progress https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.6.post1%2Bcu128torch2.7-cp39-abi3-linux_x86_64.whl'
    )
hf = bentoml.models.HuggingFaceModel(bento_args.runtime_model_id, exclude=bento_args.exclude)
openai_api_app = fastapi.FastAPI()

We pass in the BentoArgs into the bentoml package and specify the base Python image (3.12) to use for this “bento”. We further tag the requirements.txt file from our folder so that BentoML can pip install from it in Bento Cloud at deployment.

Finally, we specify our HuggingFaceModel by parsing out the parameters in BentoArgs and specify our app — surprise! It’s a FastAPI app!

To mount it:

# service.py

@bentoml.asgi_app(openai_api_app, path=’/v1')
@bentoml.service(
    name=bento_args.name,
    envs=[
        {‘name’: ‘UV_NO_PROGRESS’, ‘value’: ‘1'},
        {‘name’: ‘VLLM_SKIP_P2P_CHECK’, ‘value’: ‘1'},
        {‘name’: ‘VLLM_USE_V1', ‘value’: ‘1' if bento_args.v1 else ‘0'}, #delete this is compute capability of GPU <8.0
        {‘name’: ‘VLLM_ATTENTION_BACKEND’, ‘value’: bento_args.attn_backend},
        *bento_args.runtime_envs,
    ],
    image=image,
    labels={
        ‘owner’: ‘titus-lim',
        'type': 'prebuilt',
        'project': 'bentovllm',
        'openai_endpoint': '/v1',
        **bento_args.additional_labels,
    },
    traffic={'timeout': 300},
    endpoints={'livez': '/health', 'readyz': '/ping'},
    resources={'gpu': bento_args.tp, 'gpu_type': bento_args.gpu_type},
)
class LLM:
    hf = hf

    def __init__(self):
        self.stack = contextlib.AsyncExitStack()
        self.client = httpx.AsyncClient(base_url='http://0.0.0.0:3000/v1')

In theory a Bento is essentially a container (like Docker), but in Python it’s represented as a double decorated class:

• @bentoml.asgi_app wires up the FastAPI app.
• @bentoml.service defines deployment metadata (resources, envs, labels, etc.).
• This is where we plug in our BentoArgs object to configure the service.

Importantly, you must specify /v1 in:

1. The path of your @bentoml.asgi_app handler,

2. As a value tagged to openai_endpoint under the labels parameter of your @bentoml.service handler,

3. In the base_url parameter of self.client on initialization.

This ensures compatibility out of the box with OpenAI, because OpenAI’s base_url (the URL we ping to interact with OpenAI LLMs) always has a path of v1 . The typical vLLM app also serves it on v1.

Start up 🚀

Each class converted into a Bento must further define a few more methods including two decorated methods — @bentoml.on_startup and @bentoml.on_shutdown .

# service.py

    @bentoml.on_startup
    async def init_engine(self):
        import vllm.entrypoints.openai.api_server as vllm_api_server

        from vllm.utils import FlexibleArgumentParser
        from vllm.entrypoints.openai.cli_args import make_arg_parser

        args = make_arg_parser(FlexibleArgumentParser()).parse_args([
            '--no-use-tqdm-on-load',
            '--disable-uvicorn-access-log',
            '--disable-fastapi-docs',
            *bento_args.additional_cli_args,
        ])
        args.model = self.hf
        args.served_model_name = [bento_args.model_id]

        router = fastapi.APIRouter(lifespan=vllm_api_server.lifespan)
        OPENAI_ENDPOINTS = [
            ['/chat/completions', vllm_api_server.create_chat_completion, ['POST']],
            ["/responses", vllm_api_server.create_responses, ["POST"]],
            ['/models', vllm_api_server.show_available_models, ['GET']],
            ['/health', vllm_api_server.health, ['GET']],
            ['/ping', vllm_api_server.ping, ['GET']],
        ]

        for route, endpoint, methods in OPENAI_ENDPOINTS:
            router.add_api_route(path=route, endpoint=endpoint, methods=methods, include_in_schema=True)
        openai_api_app.include_router(router)

        self.engine = await self.stack.enter_async_context(vllm_api_server.build_async_engine_client(args))
        self.tokenizer = await self.engine.get_tokenizer()
        self.vllm_config = await self.engine.get_vllm_config()
        await vllm_api_server.init_app_state(self.engine, self.vllm_config, openai_api_app.state, args)

The @bentoml.on_startup hook spins up vLLM inside Bento’s lifecycle: it creates the engine, mounts the router, and injects state into the root app.

A vLLM router? 📡

This is where Bento’s philosophy matters: BentoML requires a single root FastAPI/ASGI app to orchestrate lifecycle hooks, health probes, scaling rules, and logging. That way, Bento can inject lifecycle hooks, health probes, traffic rules, and logging consistently across all services.

But the vLLM serve command already spins up a full OpenAI API compatible FastAPI app on its own.

If we ran vLLM’s app directly inside Bento, we’d end up with two root apps fighting for control — Bento’s and vLLM’s. That doesn’t work, because Bento needs to orchestrate its app as the single entry point for deployment.

👉 The solution is to treat vLLM not as a full app, but as a router. We import its OpenAI-compatible endpoints and mount them onto Bento’s root app (openai_api_app). That way:

• Bento still “owns” the top-level FastAPI app, satisfying its lifecycle contract.
• vLLM still exposes its familiar /v1 OpenAI-compatible routes.
• You get the best of both worlds: Bento handles deployment/lifecycle, while vLLM handles inference logic.

This is why you see openai_api_app.include_router(router) — we’re embedding vLLM into Bento’s app instead of letting it run separately.

The router defines a handful of OpenAI-compatible endpoints (completions, responses, models, health, ping).

Finally, we initialize the vLLM engine, tokenizer, and config, and bind them into the app’s state via .init_app_state().

⏻ Shut down

# service.py

    @bentoml.on_shutdown
    async def teardown_engine(self):
        await self.stack.aclose()

Bento automatically calls this when scaling down or restarting replicas, ensuring vLLM shuts down cleanly.

Heartbeat ـــــــــــــــﮩ٨ـ❤️ﮩ٨ـﮩﮩ٨ـ

# service.py

    async def __is_ready__(self) -> bool:
        resp = await self.client.get('/ping')
        return resp.status_code == 200

    async def __is_alive__(self) -> bool:
        resp = await self.client.get('/health')
        return resp.status_code == 200

Finally, we include heartbeat handlers to handle pings for healthchecks.

• __is_ready__ → readiness probe (can I serve traffic now?)
• __is_alive__→ liveness probe (is the process still healthy at all?)

📂️ The TOML and requirements file

The service does not need an if__name__=="__main__": block to spin up the application. BentoML services are launched via the bentoml command-line interface (CLI), rather than a traditional Python entrypoint.

• The requirements.txt file specifies the packages needed for the service. BentoML will pip install from this file.
• The pyproject.toml file can be a little confusing because pyproject.toml files are also used for pip installing packages. In BentoML’s case, the file specifies the input arguments to be injected into the BentoArgs class — which is then passed to the @bentoml.service decorated class.

The most important section in the pyproject.toml file is [tool.bentoml.build.args] where you define core runtime parameters:

[tool.bentoml.build.args]
name = "qwen3-8b"
gpu_type = "nvidia-h100-80gb"
tp = 1
autotune = [1, 2, 4, 8, 16, 24, 32, 40]
max_model_len = 16548
model_id = "Qwen/Qwen3-8B"
reasoning_parser = "qwen3"
tool_parser = "hermes"

• Be sure to specify the model_id parameter to match the model’s listing on HuggingFace exactly.
• Since we’re serving Qwen3–8B, we also need to find the corresponding tool parser and reasoning parser suggested (and written) by vLLM, or write them ourselves. These parsers define how the LLM interprets reasoning steps and tool calls — so picking the right one is crucial for correct behavior.
• Use the [max_model_len](https://github.com/vllm-project/vllm/issues/6641) parameter if you’re running into out of memory errors on your GPU — because some GPUs cannot fit an input context that is too long especially when running concurrent requests.
• Finally, the gpu_type parameter is the default GPU that you want Bento Cloud to use for your service. Here we’ve gone big and requested an H100 GPU 😅

Another key section is [tool.bentoml.build.args.hf_generation_config] :

[tool.bentoml.build.args.hf_generation_config]
temperature = 0.6
top_k = 20
top_p = 0.95
presence_penalty = 1.5

Here, we specify the default arguments for our LLM class. This is to avoid any initialization errors right off the bat.

The ".bentoignore" file 👻

This works just like .dockerignore or .gitignore: list the files you don’t want bundled when building your bento. (BentoML borrows many concepts from Docker, which makes it easy to pick up if you’ve used Docker before.)

Spinning up our service 💫

In your command line, run:

 bentoml cloud login

BentoML will prompt you for your API key or to create one:

Once logged in, go to the folder and run:

bentoml deploy service:LLM

💥 Got an error? Yup, that’s because I asked for an NVIDIA H100 GPU… on the free tier. Classic champagne taste on a beer budget moment 😅

To see the list of GPU instances available, run:

Since I had only $10 credits, I went with an NVIDIA L4 GPU. Just for fun, I spun up three replicas instead of one:

bentoml deploy service:LLM --instance-type "gpu.l4.1" --scaling-min 1 --scaling-max 3

Our deployment is ready!

Why not deploy an even larger model? I tried to deploy the GPT-OSS 20B but received an error message stating that the model’s size (in GB) exceeded free-tier limits.

Note: Qwen3 is not a gated model on HuggingFace. If you are using a gated LLM, do create a HuggingFace API key secret first and pass it into your deploy service command:

bentoml secret create huggingface HF_TOKEN=$HF_TOKEN
bentoml deploy service:LLM --instance-type "gpu.l4.1" --secret huggingface --scaling-min 1 --scaling-max 3

Chatting with our LLM

Ready to take our LLM for a spin? Since our deployment is OpenAI API compatible, we can just use the OpenAI SDK directly!

import os
from dotenv import load_dotenv, find_dotenv
from openai import OpenAI

_ = load_dotenv(find_dotenv())

client = OpenAI(
    base_url=f'{os.getenv("qwen3_endpoint_url")}/v1', #add "/v1"
    api_key=os.getenv("BENTO_CLOUD_API_KEY"),
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role":"user", "content":"Explan what quantum computers are and how they work like I'm a 16-year-old."},
    ],
)

Did we melt our GPUs?

Thankfully, BentoML has a collection of many useful dashboards to answer this question:

This dashboard shows that we kept to just 1 replica — so our 2 other replicas were on standby

These dashboard showed the latencies of our model — pretty fast!

Can it go faster? Yes. With stronger GPUs that we don’t have access to. But this is already not bad at all.

Thinking of what we just did

The speed to deployment perspective

We just deployed an open-sourced model on a closed-sourced platform in pure Python (yes, just one .py file) — and skipped the pain of resource prep, orchestration, and picking up new ops toys like Kubernetes, Terraform, or Ansible.

All that in under half a day’s work.

The performance perspective

vLLM is already very fast. Add robustness to your vLLM deployment and it’s chef’s kiss.

The cost perspective

Though we’re on the free tier, let’s stay real: GPUs aren’t cheap. The screenshot below shows BentoML’s hourly GPU rates.

Imagine using several NVIDIA B200s to host a Kimi K2 model (only 1 trillion parameters), and then spinning more NVIDIA B200s to serve as replicas. Poof — budget gone faster than free snacks at the office pantry.

In contrast, API calls to closed-sourced LLMs, and even open-sourced ones on vendors like AWS Bedrock are often cheaper under low traffic. I spent $1.75 total in Bento Cloud (yes, I deployed other models “for fun”), and it would’ve been even less if I’d just used OpenAI GPT models. Tokens are free in our case, but hardware isn’t — and your wallet will feel the pinch.

So under low traffic, paid APIs win on cost. But the value of the open-source path isn’t thriftiness — it’s control, customization, and data privacy. Think of it as the difference between renting a hotel room (convenient) and owning your house (painful, but it’s yours).

The monitoring perspective

BentoML gives slick dashboards, but they’re fully aggregated — so you can’t really tell who’s the office “GPU hog.” The fix? Add a LiteLLM Proxy Server layer over BentoML and issue API keys per project or per team. Instant visibility, instant accountability. Plus you can add different rate limits per API key to be kind to your wallet.

🤩 Bonus: the same trick works on closed-sourced LLM deployments too.

Do feel free to check out my article on LiteLLM for more details on how to setup the proxy server layer!

In closing

We pulled off a solid vLLM deployment — all open source, all robust, and all without ever betraying Python. The secret sauce? A cheeky little hack, plus BentoML’s generosity with free credits. Honestly, the SDK makes Python feel like it comes with superpowers.

And this is just Act One. In Act Two of our hypermodern adventure, we’ll try the same stunt on a retrieval-augmented generation setup.

Spoiler Alert: still Python, still fun, and still without spending a dime.

Disclaimer: All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.