Data Engineer Things

5 Data Ingestion Methods Every Data Engineer Should Know

How Data Actually Gets Into Your Pipeline- Full load, incremental, CDC, streaming, and API with Python code.

How I Integrated AI Into My Data Journey — And Stopped Worrying It Would Replace Me

The Tool Has Changed. The Motivation Hasn’t. Curiosity is still the only thing that matters

Data Modeling for Data Engineers: OLTP, OLAP, Inmon, and Kimball Explained

Understanding OLTP, OLAP, and the two roads your data can take after the transaction is recorded

Web Scraping with APIs: A Complete Tutorial with Real Examples

A Hands-On Python Guide to Extracting Clean Data Efficiently

How to Design a Data Pipeline for a National Event (System Design Guide)

Real-time ingestion, distributed aggregation, and fault-tolerant serving at national scale

SQL-only DLT pipelines in Databricks: what the docs don’t tell you

A practitioner’s guide to building production-grade medallion pipelines — without writing a single line of transformation Python. Yes…

Everything you need to know about Spark Structured Streaming

From its architecture, event-time processing, stateful processing to how it achieves fault tolerance.

Coding to CLAWing: AI-Native Data Engineering using OpenClaw and MCP

Integration of AI Assistant with Data

Writing Airflow DAGs that don’t fight you in production

Most Airflow performance problems are not infrastructure problems. Adding more workers or tuning the scheduler helps, but the biggest gains…

Unlocking Data Insights with Rill: A Comprehensive Guide to Streamlined Data Analytics

Harness the Power of Rill for Real-time Data Processing and Analytics

dbt Snapshots Explained: Track How Your Data Changes Over Time [dbt Series #6]

A hands-on guide to building Type 2 slowly changing dimensions with dbt and DuckDB

6 tools that keep your dbt project clean

There is more to dbt linting than SQLFluff

Passing custom variables to Databricks DLT pipelines using Asset Bundles

How to manage catalog names, schema names, and environment-specific config without hardcoding a single value — and actually sleep at night…

Deploying multi-pipeline Databricks bundles: keeping everything in sync

How to structure, correlate, and deploy a complete Databricks project — pipelines, jobs, workflows, and CI/CD — as one coherent unit. No…

Stop Staring at Cron Expressions: Airflow Just Got a Google Calendar Upgrade

Discover the missing visual calendar that finally lets you manage cron and timedelta DAGs like a human.

Forward Deployed Engineers: What Pharma needs to know about Tech’s fastest growing role

AI tools are getting better but getting them to work inside pharma’s regulated systems is a different problem entirely

Using Marquez as a lineage tool for Celery — adding the parent-run facet

This story is the second one about integrating Celery with Marquez using the OpenLineage Python package. In this story, we take the first…

Three Lessons Learned Building Data Platforms From Scratch

The case for boring tools, smaller ambitions, and building for the team you have

The Agentic Lakehouse: A Practical Enterprise Pattern for Trusted AI Agents

Introduction

AWS Glue Explained: From Raw CSVs in S3 to Queryable Parquet in Under an Hour

Crawlers, Data Catalog, and ETL Jobs Explained—with a real e-commerce pipeline as the example

Using Marquez as a lineage tool for Celery — connecting the tasks with datasets

This story is the third one on integrating Celery with Marquez using the OpenLineage Python package. In this story, we’ll explore how to…

Join Order Matters! Understanding how Modern Query Optimizers Think

Okay, I could have started this blog with a question: “Does the order of tables in a SQL JOIN matter?”

Apache Kafka in Short

Learn / Refresh about Kafka quickly

Message Queues vs. Pub/Sub: Decoding Asynchronous Communication (with SQS & Kafka)

Every Data Engineer Needs to Understand IAM - Your Access, Your Permissions, Your Responsibility

AWS IAM Is Just Like Airport Checkpoint- Who Gets In, What They Can Touch, and Why It Matters