We Didn’t Build “Another AI Project.”

We Rebuilt the Company’s Data Nervous System First.

Everybody wants AI.

Predictive analytics. AI agents. Smart assistants. Forecasting. Recommendations. “Chat with your data.”

The problem is that most companies trying to adopt AI are still fighting battles from 2007:

• Excel files living in SharePoint folders called FINAL_v3_REAL.xlsx
• legacy ERPs nobody wants to touch
• nightly batch jobs held together by hope
• reporting pipelines that break if Giuseppe from accounting goes on vacation

And honestly? That’s normal.

Recently, we worked on a cloud and AI-readiness transformation project for a mid-sized e-commerce company operating in the automotive spare parts industry. I can tell about the architectural decisions we made — because they reflect a problem many companies are facing right now:

“How do you evolve from fragmented operational systems into a modern AI-ready platform without rewriting the entire business from scratch?”

The interesting part wasn’t the AI itself.

The interesting part was why we chose certain technologies, why we rejected others, and how the final architecture became less about “moving to cloud” and more about creating a reliable data foundation for future AI systems.

Because here’s the uncomfortable truth:

If your data platform is chaos, your AI will simply become chaos at scale.

The Starting Point: A Very Common Enterprise Situation

The company’s ecosystem had evolved organically over time. Which is a polite way of saying:

“Every department solved problems independently for 15 years.”

The existing architecture looked roughly like this:

• An on-premise AS400 managing accounting and administrative flows
• AWS-hosted SQL databases powering the e-commerce platform
• A local SQL Server acting as a “bridge” between systems
• SharePoint + Excel files for operational processes
• Power BI dashboards connected directly to heterogeneous sources

Technically, everything worked.

Operational continuity was there.

But analytically? It was fragmented.

The architecture suffered from:

• duplicated data,
• manual integrations,
• inconsistent reporting,
• no centralized governance,
• limited traceability,
• and almost zero AI readiness.

The existing SQL bridge between AS400 and AWS was probably the best representation of the entire ecosystem:

A temporary workaround that became permanent infrastructure.

Which, to be fair, describes at least 40% of enterprise IT worldwide.

The assessment phase highlighted several critical issues around interoperability, governance, and orchestration.

Why We Chose a Lakehouse Architecture

One of the first architectural decisions was choosing a Lakehouse model instead of a traditional warehouse-only approach.

Why?

Because the company wasn’t dealing only with structured transactional data.

It also had:

• Excel files,
• SharePoint documents,
• semi-structured exports,
• operational files,
• future AI datasets,
• and potentially unstructured knowledge sources for LLM-based systems.

A classic data warehouse would have solved reporting.

A lakehouse solves reporting and future AI workloads.

That distinction matters.

The goal wasn’t simply:

“Make Power BI faster.”

The goal was:

“Create a scalable data platform capable of supporting analytics, machine learning, and AI agents over time.”

So we designed the platform around a layered architecture:

• centralized ingestion,
• scalable object storage,
• transformation pipelines,
• governed semantic layers,
• AI-ready consumption zones.

In other words:

We stopped thinking about dashboards.

We started thinking about data products.

Why AWS Glue Became the Orchestrator

One of the biggest pain points in the original ecosystem was orchestration.

Data movement was inconsistent, partially manual, and difficult to monitor.

So the first major technical choice was introducing Amazon Web Services Glue as the ingestion and orchestration layer.

At first glance, Glue isn’t the “sexiest” service in the cloud ecosystem.

Nobody posts inspirational LinkedIn content about ETL orchestration.

But Glue solved several real architectural problems simultaneously:

1. Centralized Data Movement

Instead of:

AS400 → SQL Bridge → AWS SQL → Reporting

We moved toward:

Source Systems → Glue → Centralized Data Lake

That eliminated intermediate dependencies and reduced operational complexity significantly.

2. Incremental Ingestion

We specifically designed ingestion pipelines around “sentinel columns” for incremental loads.

Why?

Because full reload strategies become catastrophic surprisingly fast.

Especially in e-commerce environments where:

• orders,
• pricing,
• inventory,
• and product availability

change constantly.

Incremental pipelines reduced:

• transfer costs,
• ingestion latency,
• infrastructure pressure,
• and recovery complexity.

Also fewer moments where someone accidentally reloads 200 million rows on Friday afternoon.

Always appreciated.

3. Native AWS Integration

Glue integrated naturally with:

• S3,
• IAM,
• monitoring,
• logging,
• governance,
• and downstream analytics services.

This mattered because the architecture wasn’t being designed as isolated components.

It was being designed as an ecosystem.

Why S3 Became the “Single Source of Truth”

Once ingestion was centralized, we needed a durable storage layer.

We selected Amazon Web Services S3 as the enterprise data lake.

This decision seems obvious today, but there were deeper reasons behind it.

S3 gave us:

• virtually infinite scalability,
• separation of storage and compute,
• low-cost historical retention,
• compatibility with analytics and ML ecosystems,
• and support for structured and semi-structured data.

More importantly:

It allowed us to centralize fragmented information into a single governed repository.

That became foundational for everything else:

• analytics,
• AI,
• lineage,
• governance,
• and semantic retrieval later on.

Without centralized storage, AI readiness discussions are mostly PowerPoint optimism.

Why We Chose Databricks Instead of “Just SQL”

This was probably the most strategic technical decision in the entire architecture.

For transformation and processing, we introduced Databricks as the core data engineering and analytics engine.

Could we have built transformations with simpler tools?

Sure.

But the project wasn’t only about reporting modernization.

It was explicitly designed around future:

• predictive analytics,
• machine learning,
• recommendation systems,
• and LLM-powered applications.

Databricks solved several future problems before they became production issues.

The Medallion Architecture Decision

Inside Databricks, we implemented a Medallion Architecture:

• Bronze,
• Silver,
• Gold.

This wasn’t architectural theater.

It solved real operational problems.

Bronze Layer

Raw ingestion.

Immutable source history.

Perfect for traceability and recovery.

Silver Layer

Cleaned and standardized data.

This became the operational truth layer.

Deduplication, normalization, enrichment, business rules.

Gold Layer

Business-ready datasets optimized for analytics and AI consumption.

This is where Power BI connected.

And later?

This also becomes the ideal layer for:

• feature engineering,
• embeddings,
• semantic search,
• and AI applications.

The Medallion approach gave us something the original architecture lacked completely:

Controlled data evolution.

Why Power BI Was Kept (But Repositioned)

One important architectural principle:

Don’t replace what already works for users unless absolutely necessary.

The business teams already relied heavily on Microsoft Power BI.

The problem wasn’t Power BI itself.

The problem was how it accessed data.

Originally, dashboards connected directly to heterogeneous systems:

• legacy databases,
• cloud SQL,
• Excel files,
• manual extracts.

That created:

• inconsistent metrics,
• refresh bottlenecks,
• governance issues,
• and trust problems.

So instead of replacing Power BI, we changed its role.

Power BI became a consumer of certified Gold datasets from Databricks.

This achieved several things:

• faster dashboards,
• centralized business logic,
• consistent KPIs,
• reduced direct source coupling,
• and significantly better governance.

Sometimes modernization is not about replacing tools.

It’s about removing architectural chaos behind them.

Designing for AI Before Deploying AI

This is where the project became genuinely interesting.

The company wanted future AI capabilities including:

• sales forecasting,
• recommendation systems,
• propensity models,
• and conversational AI agents.

But during the assessment, one thing became obvious:

The issue wasn’t model availability.

The issue was readiness.

The existing architecture had:

• fragmented datasets,
• no semantic indexing,
• no centralized governance,
• no orchestration,
• and no scalable ML infrastructure.

Which meant introducing AI immediately would have produced one thing:

Expensive confusion.

So the architecture itself became the AI-enablement strategy.

The Hidden Goal: Enabling RAG and AI Agents

One particularly interesting future scenario involved conversational AI systems integrated into internal workflows and customer support.

The proposed architecture explicitly considered:

• chatbot integrations,
• LLM-based assistants,
• natural language querying,
• and retrieval-augmented generation pipelines via Amazon Web Services Bedrock.

Now here’s the important part:

Most companies think RAG starts with the LLM.

It doesn’t.

RAG starts with:

• clean data,
• lineage,
• semantic consistency,
• structured knowledge,
• governance,
• and reliable retrieval pipelines.

That’s why the lakehouse mattered.

That’s why Gold datasets mattered.

That’s why orchestration mattered.

The AI layer was actually the final consequence of good architecture decisions upstream.

The Most Important Decision: Governance Over Hype

There’s a dangerous pattern in AI projects right now.

Teams jump directly into:

• vector databases,
• agents,
• copilots,
• embeddings,
• autonomous workflows.

Meanwhile the underlying data ecosystem still depends on manually updated spreadsheets.

That’s not transformation.

That’s technical debt with a demo layer.

This project deliberately prioritized:

• governance,
• orchestration,
• lineage,
• quality,
• observability,
• and scalability first.

Because production AI systems amplify both strengths and weaknesses.

If your data quality is poor, your AI simply becomes a faster mechanism for distributing bad decisions.

Final Thought

The most interesting part of this project wasn’t the AI.

It was realizing that successful AI adoption is mostly an infrastructure problem disguised as an innovation initiative.

The real transformation happened when the architecture shifted:

• from fragmented systems,
• to governed pipelines,
• to centralized data products,
• to AI-ready foundations.

The cloud migration itself was only one piece of the story.

The real goal was building a platform capable of evolving over the next decade without collapsing under its own complexity.

And honestly?

That’s probably the least glamorous — and most important — part of modern AI engineering.