Machine Learning Operations

MLOps Infrastructure at Mission Lane (Part 1)

Using BentoML to future-proof our Machine Learning Operations

This post is the first in a 2-part series. This first part describes our MLOps stack in detail, covering the motivation for switching from our previous home-grown stack to BentoML, the structure and definition of our bentos, the CI/CD process we use to build and test them, and how we deploy them into our kubernetes clusters. The second (shorter) part focuses on operational use cases, including service observability, how we’re using bentos in conjunction with Airflow to perform batch evaluations over large datasets, and some areas of interest for the future. Let’s dive in!

Introduction

Machine learning models are a critical component of the success of Mission Lane. Machine learning (ML) models inform our marketing decisions, credit-risk-based approve/decline decisions, initial credit line assignment, credit-line-increase decisions, fraud defenses (application and payments, no transaction fraud), collections and recoveries, and more. We use these models both in real-time/online decisioning (like when a customer is waiting for the outcome of their credit card application) and in batch/offline usages (we regularly evaluate models on our entire customer portfolio or on massive credit bureau extract datasets.) For any company that heavily relies on ML models, it is of utmost importance to have a robust Machine Learning Operations (MLOps) infrastructure. It is crucial that this infrastructure prevents (or at least minimizes) what is commonly called “training-serving skew.” The intentions of the data scientist with regards to feature calculations, model implementation, post-processing, or ensembling during development and validation must be preserved in production. Otherwise, the decisions that utilize ML model scores may be unreliable and won’t match the expectation of the business units that have signed off on the models’ usage based on analyses conducted prior to production implementation.

The easiest way to achieve this objective is to utilize a system in which data scientists can directly implement the model scoring pipeline without having to hand off a specification to an engineering group that is not intimately familiar with the details of the model development and validation processes. A possible concern, however, is that not all data scientists are trained as software engineers, and thus may not be as familiar with the software engineering best practices or infrastructure/devops considerations necessary to support a production environment.

For the first several years of Mission Lane’s existence, we solved the training-serving skew and shared data-science/engineering ownership problems using a homegrown platform named “the d2-stack”. This system worked well enough for a number of years, and allowed Mission Lane to grow to its current size. However, over time, it became increasingly clear that its design was limiting our future growth. For one, the system caused friction in developing and testing due to newly hired data scientist’s lack of familiarity with the idiosyncrasies of the d2-stack. It also limited support for diverse ML algorithms, caused serious scalability issues — especially for large batch evaluations — , and its monolithic one-service-for-all-models approach introduced unclear ownership and operational responsibilities.

About two years ago, we started to seriously engage with the idea of replacing the d2-stack with a more modern and flexible solution.

We heard about a new solution in this space called BentoML. At the time, it was still in an early pre-version-1.0 state, but it showed a lot of promise. We investigated more closely and quickly found that it was either meeting, or close to meeting, all of our MLOps requirements. Since then, BentoML has further “grown up” and validated our initial assumptions and hopes. Some of the BentoML features we love include:

Native Kubernetes support

• It fits neatly into our production infrastructure.
• We can take advantage of auto-scaling and dynamic resource requirements.
• It offers a container-based solution for software dependency isolation.

Broader ML algorithm support

• Previously, we could only support logistic regression and GBTs, but BentoML’s extensive framework support can support virtually any algorithm. Our past implementations had limitations such as no support for null valued features and no direct categoricals. There are no such limitations with BentoML.

Higher development and deployment velocity

• It’s easier and faster to develop new or modify existing services. Because it’s eager evaluation code, there’s no unfamiliar DSL to deal with, and BentoML supports interactive reloading.
• It’s straightforward to publish new bentos and get them deployed. Everything is CI/CD managed.

Faster model scoring

• Use of the native model objects (e.g. LGBMClassifier) is much faster than our old custom model evaluation solution.
• We can evaluate entire dataframes in batch (rather than one row at a time), taking advantage of pandas and numpy vectorization.
• BentoML supports async parallel evaluation of different models/processing (separate k8s pods for each model’s runner).

Clearer ownership

• Each model (or related model group) gets its own separate bento service. The developer of the bento has ownership of its maintenance and operations.

Multiple endpoints

• We can have unique endpoints for separate concerns, e.g. single JSON record vs. batch DataFrame input, or application stage vs. marketing stage.

Discoverability

• The built-in swagger.ai API documentation allows users to self-discover what input must be provided, and what the structure of the output is.

The first bentos at Mission Lane came online about a year ago, and we have not looked back. At the time of this writing, we have deployed 24 bentos in our production environment. Three of these see live traffic (real-time/online) as part of our decisioning flows, and all bentos are actively used on a daily basis for offline/batch scoring.

Our BentoML Stack

Fig. 1 features a high-level visual representation of our BentoML stack. Our discussion starts at the point where a data scientist has finalized¹ an ML model that they now want to make available as a production service that other systems can call to obtain model scores. The diagram references

• the data scientist’s responsibility to save the trained model artifact as a BentoML model and define the service.py,
• the role that CI/CD plays in building, testing, and publishing the bento to Yatai, and finally
• the way the service is deployed in our kubernetes environment.

Fig.1 Diagram of the BentoML Kubernetes MLOps stack (based on a diagram provided by Fog Dong, BentoML)

Bento Service Definitions

We follow a few conventions in the service definition of our bentos. Every standard bento is expected to implement the following endpoints:

**/feature_names**

This endpoint takes no input data (empty dict) and returns a dictionary of feature names. The output dictionary always contains at least one key named "service", which holds a list of minimal feature names that this bento requires in order for its endpoints to evaluate successfully.

**/score**

This endpoint is used for batch evaluation. It takes as input a pandas DataFrame of pre-calculated features, and returns a pandas DataFrame of model scores and risk groups. The typical computational steps performed in the function called by this endpoint are:

• input dataset validation (ensure all required features are present, no duplicates in the DataFrame index, etc.)
• an imputation layer (when provided) to fill in any nulls
• evaluation of one or more machine learning model scores
• post-processing such as applying a calibration, cutting risk groups, and ensembling component model scores.

Since many of these post-processing steps are common across multiple bentos, we have abstracted them into a separate library that our bentos can add to their dependencies. The ML model evaluations are always handled by dedicated BentoML runners. The other steps (pre- and post-processing) sometimes make use of BentoML runners, but often are just straight-up pandas DataFrame operations. The output of this endpoint is again a PandasDataFrame, which for network API calls presents as JSON in the format returned by pandas.DataFrame.to_dict(orient="index").

**/score_prescreen (optional)**

Bentos that will also be called for batch evaluation with marketing prescreen date expose a /score_prescreen endpoint. This input data is also in DataFrame format, but it will have slightly different feature names, and some of the normally expected features (e.g. ones dependent on the customer’s income, which isn’t available in prescreen) may not be available and require a prescreen-specific imputation.

**/score_json (optional)**

Bentos for models that will be called in an online/live decisioning context also expose a /score_json endpoint, which takes a single record as input, in JSON format. In this context, we may not have pre-calculated features available, so this endpoint takes in raw data, such as data products fetched from external providers (e.g. tradeline data from TransUnion, fraud provider attributes, and bank account transaction from Plaid) or internal transaction data from our card account system of record. Inside this endpoint’s function we make use of an internal feature extraction library named “alexandria” (which is added to the bento’s dependencies) to convert the raw input data into a 1-row DataFrame of features². We then make a direct call to the /score endpoint’s function to obtain the model scores and risk groups. The output is typically re-shaped into a generic format suitable for ingestion by our calling services.

Each of these endpoints is provided with a docstring and sample input and output data, which is displayed in the BentoML Swagger UI.

Fig. 2 Example Swagger UI for one of our bentos

Here is an example of using these bento endpoints to score a 100k-row dataframe. In this example we’re accessing the bento through our models.internal VirtualService, which routes to the appropriate bento (see Service Deployment).

In [1]: import pandas as pd
   ...: import requests

# Load the features dataframe (it contains more features than we need)
In [2]: df = pd.read_parquet("sample_dataset.parquet")
   ...: df.shape
   ...:
Out[2]: (100000, 1432)

# Obtain feature names from bento
In [3]: response = requests.post(
   ...:     url="https://models.internal.gcp.missionlane.com/isleroyale/feature_names",
   ...:     json={}
   ...: )
   ...: feature_names = response.json()["service"]
   ...: len(feature_names)
   ...:
Out[3]: 241

# Validate that all expected features are present in the dataframe
In [4]: assert all([x in df.columns for x in feature_names])
   ...:

# The dataframe is too large to send in one /score request, so let’s batch it.
In [5]: from more_itertools import sliced
   ...:
   ...: batchsize=10000
   ...: results = []
   ...: for b, idx in enumerate(sliced(df.index, batchsize)):
   ...:     print(f"Scoring batch {b}: {len(idx)} rows")
   ...:     response = requests.post(
   ...:         url="https://models.internal.gcp.missionlane.com/isleroyale/score",
   ...:         data=df.loc[idx, feature_names].to_parquet(),
   ...:         headers={"Content-Type": "application/octet-stream"}
   ...:     )
   ...:     results.append(pd.DataFrame.from_dict(response.json(), orient="index"))
   ...: df_scores = pd.concat(results)
   ...:
   ...: df_scores.shape
   ...:
Scoring batch 0: 10000 rows
Scoring batch 1: 10000 rows
Scoring batch 2: 10000 rows
Scoring batch 3: 10000 rows
Scoring batch 4: 10000 rows
Scoring batch 5: 10000 rows
Scoring batch 6: 10000 rows
Scoring batch 7: 10000 rows
Scoring batch 8: 10000 rows
Scoring batch 9: 10000 rows
Out[5]: (100000, 6)

In [6]: df_scores.head()
Out[6]:
            iguana_score  iguana_score_calibrated  ironwood_score  ironwood_score_calibrated  isleroyale_score_calibrated  isleroyale_risk_group
2784420198      0.067414                 0.157625        0.062330                   0.072288                     0.127256                      4
2471205388      0.150578                 0.449987        0.250943                   0.289916                     0.439318                     14
2862685355      0.064150                 0.150135        0.069264                   0.080534                     0.131838                      5
485026879       0.328465                 0.896285        0.313019                   0.389627                     0.704111                     20
2968403774      0.008604                 0.022660        0.007697                   0.007331                     0.015508                      1

Non-standard bento examples

In addition to our normal bentos used for online and batch evaluation of individual models or simple ensembles, we also have a few non-standard bentos that perform more complex tasks.

Auto-model bentos

At Mission Lane we use an “auto-model” process to re-fit reference models with the latest available labeled data on a daily basis. The resulting auto-models contrast with the current champion model to provide an early warning of model degradation or simply indicate how much performance boost a full model re-train might deliver. Our auto-model bentos expose a /refit endpoint, which re-fits the existing model with a dataset that is downloaded from cloud storage, and then scores this refit model on a holdout dataset. The resulting scores are written back to cloud storage, which the calling system uses to calculate performance metrics that are visually displayed in an auto-model performance dashboard. The auto-model bentos have substantially higher resource requirements than normal evaluation-only bentos, and that motivated separating them out into their own bentos, rather than just adding a /refit endpoint.

Bentos that call other bentos

We have several use cases in which we need to evaluate a host of different models on the same input data. One is our direct-mail marketing process, in which we need to evaluate multiple models that assess the likelihood of responding to a mail piece, the chances of being approved at application stage, the expected usage of our card, and others. Another is our CLIP (credit line increase program) evaluation, in which we need to evaluate multiple versions of credit risk and utilization models. Rather than orchestrating many bento API calls individually and combining their responses in the calling system, we have created “base bentos,” which handle these evaluations and aggregate the results. The base bento either receives the evaluation dataset in the request payload or loads it from cloud storage, and proceeds to make network calls to each of the required bentos. These network calls are made asynchronously (using concurrent.futures), which further speeds up the process. The base bento supports multiple retries, in case any of the network calls to the child bentos fail unexpectedly.

The bentoml-models repository

Our bentoml-models github repository (repo) holds the definitions of all of the bentos at Mission Lane. Every bento is defined entirely in its own sub-directory, and no references to any other parts of the repo are permitted. The repo has a directory structure that mirrors our business domains (see Fig. 3 left).

Fig. 3 Directory structure of the bentoml-models repo (left), Typical contents of a bento directory (right)

Contents of a typical bento directory (see Fig. 3 right):

• import_models.py — A python script that downloads model artifacts from cloud storage and constructs the bento models from them
• service.py — Contains the python code defining the computations performed by the bento’s endpoints
• model_definitions.py — Holds feature names, calibration parameters, bin edges, etc. for each of the component models
• bentofile.yaml — Defines the modules included with this bento and its software dependencies. Some of our dependencies are internal modules, which the bento needs to pull from our artifactory service. To support this we use a custom base image, which starts from python3.10 and includes the artifactory credentials as environment variables.
• publish.yaml — This file determines to which Yatai environments the CI/CD will publish this bento.
• requirements.txt — This file specifies the development software environment. This is purely for convenience, and is not used in the construction of the bento itself.
• Dockerfile.cicd — This file specifies the Docker container that is used by CI/CD to build the bento, run the unit tests, and publish to Yatai.
• tests/ — This directory contains unit tests, expectations, and resources.
• sample_data/ — This directory contains input and output sample data to be displayed in the swagger UI.

Development

One of the advantages of the BentoML ecosystem is how easy it is to develop bentos. At Mission Lane, the bento development process starts by gathering the model artifacts (the trained model and any associated transformation pipelines) and establishing the exact software versions that were used during development. Since building the bento is usually the responsibility of the data scientist that developed the model, these tend to be readily available. The artifacts are uploaded to a standardized cloud storage location. The data scientist then creates a python virtual environment for code development and local testing, and installs the specified versions of any required libraries (e.g. sklearn, XGBoost, LightGBM, etc.), the BentoML package itself, and any other auxiliary packages that might be helpful (e.g. ipython, jupyterlab, etc.). The developer then creates the import_models.py script, which loads these artifacts and saves them as bento models to the local bento repository.

During the creation of the service.py file it is highly advantageous to have the bento service running in development mode: bentoml serve ./service.py:svc --reload. This command starts a locally running bento service (localhost:5001), which automatically reloads anytime a change to service.py is made. The developer can then repeatedly make API requests against this service and immediately observe the results of their edits. This allows for rapid iteration and progress.

Once the service has been finalized, the bentoml.yaml file is created. Here is an example:

# bentofile.yaml
service: "service:svc"
include:
 - service.py
 - model_definitions.py
 - transformers.py
 - sample_data/
python:
  packages:
   - alexandria==0.3.10
   - lightgbm==3.3.2
   - pandas==2.0.3
   - pyarrow==11.0.0
   - scikit-learn==1.1.1
  extra_index_url:
    - "https://artifactory.pennywise.cc/artifactory/api/pypi/python/simple"
docker:
    base_image: "docker.pennywise.cc/data/custom_bento_base:0.0.12"

In this example, the bento requires the Mission Lane internal alexandria feature extraction library, which is obtained from our private pypi service specified under extra_index_url. All of our bentos utilize a custom base image, which is hosted in our docker image repository and includes a few pieces of software (e.g. libgomp1, jq, etc.) as well as credentials for connecting to our pypi server.

The final step is to create sample input and output data for each of the endpoints (to be displayed in the swagger UI) and to create unit tests for each of the service endpoints. The data scientist will then raise a PR for review, and once the bento is merged it will be built and published to Yatai via CI/CD (see below).

Yatai

Yatai is a centralized bento (and model) repository that is part of the BentoML ecosystem. Its web UI allows users to discover information about bentos and their deployments (see Fig. 4) and allows us to keep track of bento versions, which models they include, and when and by whom they were published. When a user needs access to a bento locally (e.g. on their laptop), they can pull it down from Yatai into their local bento store. For deployed bentos, Yatai can be used to obtain the k8s cluster internal URL, get information about the k8s pods (e.g. how runner names map to k8s pod names), and even inspect the logs of individual pods.

Yatai is responsible for building the bento docker image, pushing it to our internal Docker image repository, and deploying it into our k8s clusters. I discuss deployment in more detail in Service Deployment. We have set up Yatai in a traditional 3-environment system: development (dev), staging, and production (prod). In the development environment, we allow users to publish bentos to Yatai-dev directly via the BentoML command-line interface. In staging and production environments, however, we require CI/CD. This is enforced by role-based access control (RBAC): the Yatai user roles in staging and production do not permit pushing bentos (or deploying them).

Fig. 4 Screenshot of the Yatai UI, showing a number of published and deployed bentos.

CI/CD

Git-ops and infrastructure-as-a-code (IaaC) are two key principles at Mission Lane engineering, because of the increased stability, reliability, and productivity resulting from automated, codified, and repeatable infrastructure. Accordingly, we have tight integration with CI/CD (Github Actions) to build, test, and publish bentos, as well as for deployment (see Service Deployment). When a new set of commits is pushed to github (or when a PR is merged to main), CI/CD performs the following steps (see Fig. 5):

1. Run a linting and code style check. We use brunette (a variant of black), isort, and flake8.
2. Determine which bento directories (under bentos/) have changes compared to main’s HEAD.
3. For each changed bento directory, and running in parallel:

• Build the bento’s CI/CD Docker image. The base image (custom_bento_base_cicd) is derived from the base image that is used for the bento itself, and includes additional packages needed to build the bento and run the tests. In the CI/CD container we pip install each of the bento’s dependent python packages, copy over the bento’s resources, import the bento’s artifacts (RUN python ./import_models.py), and then build the bento (RUN bentoml build). The resulting container holds the bento and its component models, and is fully capable of executing any bentoml command.
• Run the bento’s unit tests, using the Docker image: docker run --rm ${{needs.build.outputs.image-full-name}} python -m pytest ./tests
• Our unit testing approach makes use of the bento that was built inside the container. We load the bento locally, initialize the runners, and then call the API methods directly³. For example:

bento_service = bentoml.load("<BENTO_NAME>:latest")
[x.init_local() for x in bento_service.runners]

def test_score_dataframe():
    df_scores = bento_service.apis["score"].func(df_features)
    assert_frame_equal(df_scores_expected, df_scores)

• On merge-to-main, we publish the bento to Yatai in each of the environments specified by the publish.yaml file:

docker run --rm ${{ inputs.image }} /bin/bash -c \
            "bentoml yatai login --api-token $YATAI_API_TOKEN --endpoint $YATAI_ENDPOINT;
             bentoml push ${{ inputs.bento }}:latest"

The fully parallel nature of our github actions CI/CD pipeline means that our runtimes are acceptable even when many bentos are affected. For example, we recently upgraded the bento base image in all of our bentos. The resulting CI/CD run consisted of 26 parallel runs of the pipeline described above, resulting in new bentos being pushed to Yatai in all three environments, and the whole run completed in 12m40s.

Fig.5 Screenshot of our github actions CI/CD pipeline, triggered by a PR merge.

Service deployment

BentoML’s native kubernetes integration makes it easy to deploy new services. Once a bento has been published to Yatai, it can be deployed as a microservice in our kubernetes clusters. This deployment process is managed by Yatai. The details of Yatai’s architecture are described here. In brief, Yatai consists of several components running in kubernetes:

• yatai-image-builder is responsible for building the bento’s Docker image and publishing it to our image repository. It does this using a kubernetes Custom Resource (CR) named **BentoRequest. After building the image, the BentoRequest generates a second CR named [Bento](https://docs.yatai.io/en/latest/concepts/bento_crd.html)**, which describes the bento image and its runners.
• yatai-deployment is responsible for deploying the bento service. It makes use of a **BentoDeployment** CR, which specifies the computational resources (cpu/memory requests and limits for service and runners, as well as replicas and auto-scaling configuration,) creates the pods, and launches the deployment.

In practice, each bento’s BentoRequest and BentoDeployment are specified by Custom Resource Definitions (CRD) that are version-controlled in our bentoml-config github repo. Application deployments at Mission Lane are fully controlled by GitOps using ArgoCD. We have defined a kustomize ArgoCD application named “bento” which allows us to generate and ensemble the various kubernetes manifests that make up the full bento service in all three environments (dev, staging, and prod.) Besides the bento-native BentoRequest and BentoDeployment CRs, this includes VirtualServices that expose our bento endpoints outside of the kubernetes cluster, AuthConfig and SealedSecrets for API authentication, cloud StorageBuckets and associated access control, and Prometheus error alerting rules.

The typical process to get a new bento deployed is:

1. Obtain the latest version of the bento to be deployed from Yatai, either via GUI or API, e.g.: isleroyale:6q5hyftjfostszqw.
2. Create a directory under bentos/k8s/{dev,staging,prog}/ for the new bento.
3. In this directory, create a bento.yamlfile containing the CRDs for BentoRequest and BentoDeployment. In dev we typically just have a single replica by default (minReplicas: 1) and quite low values for the requests (cpu: 30m, memory: 600Mi), but in staging and prod we are more generous (min/maxReplicas: 2/10; cpu req/lim: 2000m/4000m; memory req/lim: 1Gi/12Gi). (Of course these are tuned to the particular use cases.)
4. Add an entry in bentos/k8s/{dev,staging,prod}/kustomization.yaml in order to include these new k8s manifests.
5. Add a yaml file under .cortex/catalog/ describing this bento and providing links to its swagger docs, so that this bento service can be discovered in our Cortex.io service catalog.
6. Optional: add an entry in the istio VirtualService manifest, if this bento needs to be accessed outside of the kubernetes cluster.

To deploy an update to an existing bento (a new version, or k8s resource changes) we only need to modify the bento.yaml file.

Once a bento has been deployed, it can be accessed in a number of different ways:

• Within the cluster: Other services running in the kubernetes cluster can access the bento directly at http://<bento_name>.yatai.svc.cluster.local:3000. It is possible to port-forward the service (kubectl port-forward) from the cluster to a port on a laptop.
• Outside of the cluster, but within our VPN: Personal laptops and services or VMs running outside of our kubernetes cluster can access the bento through our istio virtual service at https://models.internal.gcp.missionlane.com/<bento_name>/, provided that a routing for the bento has been added (step 6 above).
• From the internet: In some circumstances, we need our bentos to be accessible to external partners such as a SaaS decisioning system or marketing platform. We support this through our universal API gateway at https://api.missionlane.com/models/<bento_name>/, but it requires additional configuration. The external partner’s IP addresses must be yes-listed, and we require API authentication in the request header.

BentoML is open source!

The flexibility, scalability, and automation that BentoML has afforded the MissionLane data science and engineering teams was enabled by it being open source⁴. Had it not been open source, it would have been difficult to gain the confidence to really explore BentoML as a viable option for replacing our aging MLOps stack, and we would have probably settled for an inferior commercial product. The open source nature of the project has engendered a large community of contributors that are working to improve the system, and an active public Slack workspace in which problems are discussed and solved.

We are proud to say that Mission Lane employees have directly contributed feature enhancement and bug fix pull requests, submitted numerous GitHub issues to both BentoML and Yatai, and are active contributors on Slack. We fully intend to expand these contributions to the BentoML project in the future.

Nothing is perfect…

If you’ve read this far, then you understand that we’re big fans of the BentoML platform. Its adoption has resulted in major improvements to our MLOps systems. But of course nothing is perfect, and there are areas for improvement. We list some of these below, with the implicit understanding that these are not criticism of the BentoML team, but rather opportunities for contributions:

• There is no SSO/OIDC for user management in Yatai. This means that user creation and permissions must be manually managed by a human admin. (Yatai Issue #259)
• It is not currently possible to assign default k8s tolerations that automatically apply to both service and all runner pods. (This is currently being worked on by the BentoML team.)
• We have never been able to observe any logs from runners. We occasionally find that the service errors because something went wrong in one of the runners, and it would be helpful to see its logs.
• There is no native support for GET endpoints. Our /feature_names endpoint really shouldn’t have to be a POST, since we’re not passing any information in. BentoML can support GET endpoints by bundling with ASGI or WSGI apps (see here), but it requires using FastAPI or Flask in addition to BentoML.
• To update the bento version of an existing deployment, we have to manually restart the Yatai-deployment component for the new pods to be deployed. To avoid this, we’ve been following a temporary solution in which we change the name of the BentoRequest CR, and correspondingly the bento field in the existing BentoDeployment CR, any time we update a bentoTag. This creates a new BentoRequest CR, which then triggers the BentoDeployment to deploy new pods. This trick works, but it’s a workaround, at best.

To be continued…

This concludes part one of our 2-part series describing Mission Lane’s MLOps Infrastructure. In the second part we discuss some impressive operational use cases, including how we’re using bentos in conjunction with Airflow to perform batch evaluations over large datasets.

Stay tuned!

Acknowledgements

We would have never gotten to the current state of our BentoML integration without invaluable contributions from many people. On the BentoML team I would like to especially call out Chaoyu Yang, Bozhao Yu, Tim Liu, and Sean Sheng. Thank you for building BentoML! On the Mission Lane side (current and past) I would like to acknowledge: Stan Bartlett, Joe Bond, Chris Cureau, Alex Daidone, Alex Hasha, Rajat Jatana, Hans Knecht, Esteban Quevedo, Kirill Stolz, plus numerous Mission Lane data scientists and MLEs that have actually built and deployed bentos and worked on the Airflow DAGs. A special thanks to Steve Stevenson and Lani Allen for editing this article.

Footnotes

¹ The model development pipeline itself, i.e. the training of the models, is not in scope of this article, but will be described in another blog post.

² In the future we are planning to delegate all feature calculations to a feature store service (chalk.ai), rather than performing them inside the bento. At that point we will deprecate the /score_json endpoints in favor of just calling /score with a 1-row features dataframe in a live decisioning context.

³ Our method of unit testing bentos predates some BentoML developments that make it easier to programmatically serve and interact with bentos. https://docs.bentoml.org/en/latest/concepts/bento.html#via-bentoml-server-api

⁴ BentoML also has a commercial offering called BentoCloud, which “provides fully managed infrastructures for deploying BentoML, OpenLLM, or any model, optimized for performance, scalability, and cost-efficiency.” We don’t have any first-hand experience with it.