Running a Real‑Time Scoring Service: Comparing Best‑of‑Breed MLOps to Vertex AI

Introduction

At mobile.de, we build products that help professional car dealers make better decisions faster. Dealers operate in a high‑variance environment: inventory turns quickly, demand shifts by region and season, and pricing and lead volumes can change dramatically over short periods. Turning that complexity into reliable analytical solutions means more than building accurate models. It means delivering predictions in a way that is fast, dependable, and easy to evolve as the market changes.

One example is a production scoring service that estimates the likelihood of a sales outcome for a listing within a defined time window (internally, we refer to this as a “sales probability” score). The service runs in two modes:

• Online: a low‑latency endpoint used in real‑time decision flows
• Batch: scheduled scoring for backfills and offline consumers

We ran a proof of concept (PoC) to evaluate whether Vertex AI, now part of Google Cloud’s Gemini Enterprise Agent Platform, could replace our best‑of‑breed MLOps setup end‑to‑end. Specifically, we compared a stack built from specialized components — Airflow for orchestration, self‑hosted MLflow (VM + Postgres) for experiment tracking (artifacts in GCS), and online serving as Docker containers deployed as pods on a Kubernetes cluster on AWS — against a more consolidated approach using Vertex AI for orchestration, tracking, and online serving.

Why this PoC?

The motivation was simple: shipping and operating ML products is rarely blocked by modeling but by the operational glue around it.

For dealer‑facing analytics, operational friction becomes customer friction:

• Reliability matters: a flaky scoring service erodes trust quickly.
• Iteration speed matters: markets shift; models and features must keep up.
• Safe rollouts matter: we need predictable deployment and rollback paths.

In our setup, the MLOps “glue” spanned multiple tools and (importantly) multiple ownership boundaries. The PoC asked:

Can we reduce dependencies on other development teams by moving from a best‑of‑breed toolchain to Vertex AI as a single platform for the ML lifecycle?

Dependency reduction in this case means optimizing for a smoother, more autonomous loop for the team that owns the model:

• fewer handoffs for DAG/pipeline changes
• fewer operational tasks to keep tracking infrastructure healthy
• fewer bespoke deployment steps for online serving

Key takeaways

• Consolidation is real: Vertex AI can replace Airflow orchestration, the MLflow VM, the Postgres backend and it can take over online serving in place of Kubernetes‑based serving on AWS.
• Cost shifts: experiment tracking can be cheaper with Vertex AI (based on our estimates), but online serving becomes significantly more expensive in our setup.
• Velocity trade‑off: recurring workflow run times stayed stable, while deployment to Vertex endpoints was slower in the PoC.
• Production experimentation improves: Vertex AI’s traffic splitting and multi‑model endpoints make controlled rollouts and A/B‑style testing easier.

(Exact time and cost depend heavily on region, traffic patterns, scaling configuration, and your operating model. This should be treated as a decision aid, not a universal benchmark.)

The ML project lifecycle

ML systems aren’t “train once, deploy forever.” A typical ML lifecycle includes repeated loops of:

1. building features and training
2. tracking experiments and selecting candidates
3. validating models (offline + sanity checks)
4. deploying to production
5. monitoring performance and drift
6. retraining and iterating

Once a model goes into production, non‑modeling requirements become first‑class concerns, especially for an online endpoint:

• latency and availability
• error handling and fallbacks
• observability (what model ran, on which data, with which config?)
• safe rollouts and rollback paths

This is where MLOps choices matter not just for getting a model to run, but for keeping the service reliable, auditable, and easy to evolve.

The baseline: best‑of‑breed MLOps (what we run today)

Our current setup is a practical “compose what works” architecture:

• Airflow (Cloud Composer) handles orchestration.
• MLflow runs on a self‑hosted VM for experiment tracking.
• Postgres (Cloud SQL) is used as the MLflow backend store.
• Model artifacts are stored in GCS.
• Online serving runs as Docker containers deployed as pods on a Kubernetes cluster on AWS.

current architecture diagram

This stack is powerful, but it can accumulate operational handoffs across teams (for example: orchestration support, tracking infra maintenance, and deployment tooling).

The alternative: Vertex AI as the “one platform” option

Vertex AI (now branded under the Gemini Enterprise Agent Platform) offers an integrated platform covering major MLOps building blocks end‑to‑end, especially if you want one place for:

• pipeline runs
• experiment tracking / metadata
• model registry
• deployment and rollout controls

Vertex AI architecture diagram

For this PoC, the target state was intentionally bold: replace multiple moving pieces with Vertex AI equivalents and assess whether this would materially reduce operational dependencies.

Changes vs constants: what moved, what stayed

We kept some parts intentionally constant to make the comparison cleaner (for example, artifacts staying in GCS). The main changes were:

What gets replaced

• Airflow orchestration → Vertex AI scheduled pipelines
• MLflow (VM) experiment tracking → Vertex AI experiments tracking
• Postgres MLflow backend → removed
• Online serving on AWS Kubernetes → Vertex AI endpoint (online serving)

What stays the same

• GCS artifacts → GCS artifacts

Does it resolve the main pain points?

The headline goal was dependency reduction, with fewer cross‑team handovers to keep the ML system moving.

Vertex AI helps with infrastructure consolidation, but consolidation is not “free.” Effort often shifts into:

• platform configuration and IAM
• Vertex‑specific deployment mechanics
• different cost and scaling models

A useful mental model is to ask: Which pain points disappear, and which ones relocate?

To make that concrete, we recommend mapping pain points to outcomes (improves / neutral / worsens). For example:

Comparison scorecard: time and cost

Methodology

We focused on two pragmatic dimensions:

• Time: recurring workflow runtime and deployment time (end‑to‑end “from trigger to ready”)
• Cost: directional monthly averages for always‑on components

We treated these numbers as directional, which is useful for early decisions, but not a substitute for a full cost model with real traffic patterns, region‑specific pricing, and committed‑use discounts.

Workflow time comparison

• Recurring orchestration time remained roughly stable across legacy and Vertex workflows (~15 minutes per run in this case).
• Vertex endpoint deployment was notably slower (~20 minutes) compared to the AWS/Kubernetes deployment flow (~5 minutes).
• Initial setup overhead exists in both worlds and should be separated from per‑run execution time.

Cost comparison (estimated monthly averages)

• Experiment tracking costs were lower with Vertex AI compared to the MLflow VM (in our estimate).
• Online serving costs increased significantly with Vertex AI endpoints (in our estimate).
• Costs related to stored artifacts on GCS remained the same.

Gains and losses: what we learned

Gains

• Infrastructure consolidation: Vertex AI can replace Airflow, MLflow VM, the Postgres backend and it can take over online serving.
• Lower experiment costs: Vertex experiment tracking was cheaper.
• Multi‑model endpoints: several models can co‑exist on the same endpoint, improving utilization.
• Flexible traffic splitting: enables controlled rollouts and efficient A/B testing between model versions.

Losses / trade‑offs

• Higher serving costs: online serving costs increased significantly using Vertex AI endpoints in our setup.
• Slower deployment: endpoint deployment time was notably slower.

Neutral

• Stable workflow times: recurring workflow runtimes remained stable across both setups.

Recommendations (what we’d do next)

Based on this PoC, the decision is less “Vertex yes/no” and more:

Use Vertex AI where it reduces handoffs and operational load without introducing disproportionate cost or friction.

Concretely:

1. Adopt Vertex AI where consolidation clearly reduces operational dependencies. Scheduled pipelines and experiment tracking are strong candidates
2. Treat online serving as a separate decision, because it can dominate cost and affects deployment velocity
3. Lean into Vertex strengths intentionally. Multi‑model endpoints and traffic splitting are only valuable if your release process is designed to use them
4. Re‑run the evaluation with tighter measurement, with realistic traffic patterns, CI/CD automation, a clear ownership model and region‑specific pricing.

Closing

This PoC showed that “one platform” is not just a tooling choice but an operating model choice.

Vertex AI can simplify the MLOps surface area and reduce the number of systems to own. But especially for online serving, that simplicity may come with higher cost and slower deployments.

The best outcome is a deliberate platform decision: consolidate where it removes friction, and be selective where it introduces new constraints.