#data-lake

AI

ME Mehul Batra

Part 2 — The Deconstructed Database: Building a AI+BI Lakehouse from Open Parts

How to assemble a query layer over a lakehouse that holds both tabular and AI-native data, and serve analytical, full-text, and vector…

Jul 20, 2026 · 11 min read · 0 claps #ai #apache-iceberg #lance

TCH

JO Jonathan Saring · Level Up Coding

7 Managed Iceberg Lakehouse Solutions You Should Know

Apache Iceberg is the default table format for the modern lakehouse. In the State of Apache Iceberg in the Enterprise (2026) survey of 252…

Jul 20, 2026 · 15 min read · 47 claps #data #data-engineering #software-development

ART TCH MKT

SU Sudibyo Dwicahyo

Data Warehouse di Era Modern Big Data Architecture: Masih Relevan atau Mulai Tergantikan?

Pertanyaan ini muncul ketika dari postingan sebelumnya menyebutkan bahwa Data Lakehouse merupakan pendekatan modern terkait kebutuhan…

Jul 20, 2026 · 6 min read · 0 claps #data-analytics #big-data #data-architecture

ART TCH

PR Prem Vishnoi(cloudvala) · EndToEndData

AWS Data Lake Cost Optimization: A Practical Guide for Data Engineers and Architects

Introduction:

Jul 18, 2026 · 4 min read · 51 claps paywalled #aws #data-lake #cost-optimization

TCH

VI Vinicius Morais · InfoSec Write-ups

Advanced security alerting with DuckDB and Delta Lake | Clickdetect

Hey, do you like DuckDB and want to generate security alerts using it? I have a solution for you.

Jul 17, 2026 · 4 min read · 5 claps #delta-lake #duckdb #siem

ART

MU Muthupalaniappan Ramanathan

We Saved Millions Without Optimizing a Single Query

The architectural decision that changed how we thought about data platforms forever.

Jul 14, 2026 · 3 min read · 0 claps #data #data-architecture #data-lake

TCH SCI AI

MI Milan Bins Mathew · MiQ Tech and Analytics

How MiQ uses AI to cut storage costs at petabyte scale

Milan Bins Mathew, Software Engineer II, Data and AI Platforms, MiQ

Jul 14, 2026 · 5 min read · 2 claps #databricks-genie #zstandard #storage-cost-optimization

TCH AI

RA Raj R. Sharma

The Secret Behind Fast Data Lakes: Apache Parquet

When working with modern data lakes, storage efficiency and query performance are two of the biggest challenges. As datasets grow from…

Jul 12, 2026 · 4 min read · 0 claps #data-lake #apache-parquet #data-analysis

TCH SCI

AR Arzu

Before AI, You Need Data: Understanding Data Warehouses and the Modern Data Stack

A Physics Engineer’s Journey into Data & AI

Jul 12, 2026 · 10 min read · 0 claps #database #data-warehouse #data-lake

TCH SPT

A Алексей Выродов

Running Apache Cloudberry on OpenShift/OKD4 the Right Way — A Full End-to-End Walkthrough

A production-grade CloudberryCluster on OKD4 with a scoped custom SCC (not anyuid), Vault-PKI TLS, Keycloak OIDC, and MinIO S3 backups —…

Jul 11, 2026 · 24 min read · 0 claps #apache-cloudberry #mpp #data-lake

ART TCH

ME Mehulbudasana

Data Lake vs Data Fabric: Which One Does Your Enterprise Actually Need?

Introduction

Jul 07, 2026 · 7 min read · 0 claps #data-lake #data-fabric #comparison

CA Carina Mendes · TOTVS Developers

Cloud FinOps na prática: como transformei monitoramento mensal em gestão ativa de custos em uma…

A escala de demanda de uma plataforma de dados vem junto com o bônus da adoção, mas também carrega outros ônus, como o aumento de custos…

Jun 23, 2026 · 10 min read · 2 claps #data #cloud-finops #plataforma-de-dados

TCH

AH Ahmed M Khan

Understanding Slowly Changing Dimensions (SCDs) in Data Warehousing: Types 0, 1, 2, 3, 4, 5, and 6

When building a data warehouse, one of the most common challenges is handling changes to business data over time.

Jun 21, 2026 · 5 min read · 1 claps #data-warehouse #data-engineering #slowly-changing-dimension

TCH

MU Muhammad Umar · Towards Data Engineering

Data Lake: 50 Data Feeds, Zero Standards, and the Framework That Fixed It All

We Had Shell Scripts for our hadoop platform and Prayers. Here’s the Ingestion Framework We Built Instead.

Jun 18, 2026 · 8 min read · 3 claps #data-engineering #data-lake #kafka

AI TCH

MA Maximilian Oliver · AWS Tip

Building Autonomous Data Lake Governance With AWS Glue, Lake Formation, and Generative AI

How I created self-governing data lakes that automatically classify data, enforce policies, detect quality issues, and use generative AI to…

Jun 10, 2026 · 5 min read · 0 claps paywalled #generative-ai-tools #ai #data-governance

TCH

PU Purvajage

Understanding Data Lake vs Data Warehouse vs Data Lakehouse

During my Data Engineering internship, I started working with tools like HDFS, Jupyter Notebook, and Databricks while building data…

Jun 09, 2026 · 4 min read · 0 claps #data-engineering #data-lake #data-warehouse

ECO TCH AI

NA Narwal

Unlocking Confidence in Unstructured Data: Addressing Top Challenges in the Data Lake Ecosystem

As organizations expand their digital transformation efforts, they increasingly rely on data lakes for unstructured data storage and…

Jun 09, 2026 · 2 min read · 0 claps #unstructured-data #data-lake #data-integrity

MDA TCH GEN MKT

SA Saurav Singh · Towards Data Engineering

Delta Lake on Azure: What the Docs Don’t Tell You

VACUUM deleted files while a live stream was still reading. Schema evolution broke things it wasn’t supposed to touch. Here’s what…

Jun 08, 2026 · 6 min read · 50 claps paywalled #azure #databricks #data-engineering

HE Hector Poblete

Iceberg — Formato de Tabla

Las organizaciones que son capaces de generar y obtener volúmenes importantes de datos necesitan de una forma eficiente para almacenarlos y…

Jun 07, 2026 · 4 min read · 0 claps #data-engineer #data-lakehouse #apache-iceberg

TCH

SR Sriw World of Coding · T3CH

Cracking the Ultimate PySpark Interview Question: How to Flatten Dynamic JSON Without Hardcoding…

How to flatten a completely dynamic, semi-structured JSON column in PySpark without hardcoding a single schema or melting your cluster

Jun 07, 2026 · 7 min read · 53 claps paywalled #data-engineering-101 #pyspark #data-lake

TCH MKT

MA Mark Anderson

Why StarRocks Is Better Than DuckDB for Data Lake and Apache Iceberg Analytics

If there is one use case that made DuckDB famous, it is this one: point SQL directly at Parquet files on S3 — no warehouse, no loading, no…

Jun 07, 2026 · 10 min read · 0 claps #starrocks #duckdb #data-lake

GEN

RE Renu Tewari

Musings on Data, Part 1: lakes, houses, clouds, etc.

Renu Tewari

Jun 01, 2026 · 5 min read · 0 claps #data-lake #data-lakehouse #cloud-data-warehouse

AI TCH ECO HUM GEN

AL Alexander Makeev

From rank 46 to rank 3: training an embedding model for Data Lake table search

How we fixed table search for a banking Data Lake by teaching embeddings what “right table” means

May 31, 2026 · 22 min read · 21 claps #data-lake #llm #embedding

ART TCH AI

JS Jsanca

From Inverted Indexes to Lakehouses: A Search Engineer’s Guide to Databricks

The Vocabulary Problem

May 31, 2026 · 13 min read · 0 claps #databricks #information-retrieval #medallion-architecture

ART TCH HUM

PA Padmakypu

Why Lakehouse Architecture Feels Like the Future of Data Engineering

Recently, while learning Databricks, I came across the term “Lakehouse Architecture” almost everywhere. Initially, it sounded like another…

May 26, 2026 · 1 min read · 10 claps #data-engineering #databricks #big-data

AI TCH

ME Mehul Batra

A field journal on Ray Data and Daft for multimodal data lake

What Use Cases on KubeRay Taught Us

May 25, 2026 · 13 min read · 5 claps #ai #multimodal #data-lake

TCH

SH Sher Islam · Towards Data Engineering

How Data Warehouse, Data Lake, and Data Lakehouse Actually Differ

Introduction

May 23, 2026 · 2 min read · 4 claps #data-lake #data-engineering #data-lakehouse

TCH AI

SR Sriw World of Coding · Cubed

Data Engineer Interview Question : How to Design a Scalable Data Lake on S3 (Step‑by‑Step Guide)

Imagine your company’s data exploding every month — logs, clickstreams, user events, transactions, and more. Without a plan, you’ll end up…

May 23, 2026 · 7 min read · 51 claps paywalled #simple-storage-service #data-lake #aws

TCH AI SCI

SU SURAJ KUMAR SONI

Data Warehouse vs Data Lake: Understanding Modern Data Storage Systems

In today’s digital world, data has become one of the most valuable assets for businesses. Every company generates huge amounts of…

May 21, 2026 · 5 min read · 0 claps #data-warehouse #data-lake #data-analysis

AI TCH SCI

RY Ryan Williamson

2026 Guide to Building an AI-Ready Data Lake

A growing number of enterprises have graduated from experimenting with artificial intelligence to productionizing AI models across their…

May 18, 2026 · 3 min read · 0 claps #data-lake #data-lakehouse #ai

AI GEN

AL Alon Wolenitz

Before You Build a Context Lake

A few things worth knowing about how real ones stay fresh and why you might be building a swamp.

May 18, 2026 · 6 min read · 0 claps #ai-agent #context-engineering #data-lake

ART TCH

TR Tricon Infotech

From Data Lake to Profit Center: A Practical Guide for Enterprise Leaders

A data lake was supposed to solve everything. Store all your data in one place, in any format, at any scale, and figure out what to do with…

May 18, 2026 · 4 min read · 0 claps #data-lake #enterprise-technology #enterprise-architecture

ART MDA MKT AI TCH SCI

GE Geek Nomad · Level Up Coding

Medallion Architecture for Data Analytics: A Practical Guide to Building Trusted Data Products

A practical guide to designing Bronze, Silver, and Gold data layers for reliable analytics, trusted dashboards, and real-world data…

May 18, 2026 · 14 min read · 36 claps paywalled #data-science #big-data #software-architecture

ART TCH MKT

LE Leo Santos

Building a Serverless Data Analytics Pipeline on AWS: A Practical Reference Architecture

Traditional data platforms often feel like a bottleneck. Getting new data onboarded or spinning up fresh analytics pipelines usually means…

May 17, 2026 · 3 min read · 0 claps #aws #serverless #big-data

AI GEN

SA Santosh Sharma

Your Kafka Pipeline Doesn’t Need an LLM — It Needs AI Agents That Actually Understand Production…

Over the last year, I’ve seen countless demos showing AI agents doing magical things with data platforms.

May 17, 2026 · 9 min read · 1 claps paywalled #agentic-ai #data-lake #kafka

AI SOC

SA satyabrata mishra

AI-Ready Healthcare with Oracle Autonomous AI Lakehouse

How Hospitals Can Collaborate Securely Without Moving Sensitive Data

May 15, 2026 · 3 min read · 0 claps #ai-in-healthcare #oracle-database #autonomous-database

ECO

GR Graylog · The Visibility Layer by Graylog

How 6 Months of Log Retention Can Save You $1 Million

Kimber Spradlin, Chief Marketing Officer at Graylog

May 15, 2026 · 8 min read · 0 claps #log-management #graylog #data-retention

ART TCH AI

VI vidyasagar chokkara |devsaga22

Data Engineering — 1: An Eagle Eye View of the Modern Data Landscape

In simple term Data Engineering is the process of converting and serving available data into a consumable form i.e curated data for the…

May 10, 2026 · 8 min read · 7 claps #medallion-architecture #data-engineering #microsoft-azure

TCH

MA Manoj Kumar

Tools Behind Data Engineering - Mapped to the Data Pipeline

A practical breakdown of the tools data engineers actually use — and where each one fits in a real system

May 01, 2026 · 3 min read · 1 claps #data-engineering #data-ingestion #data-lake

TCH MDA

CO Constantin Lungu · Data Engineer’s Notes

BigQuery BigLake Tables Explained: What They Are and When to Use Them

If you’ve worked with BigQuery external tables before, you know the basic idea: a thin wrapper around data that resides somewhere else, but…

Apr 26, 2026 · 4 min read · 3 claps paywalled #bigquery #sql #data-engineering

ECO TCH

MS MS

Transforming Data Management with Secure and Scalable Data Lake Solutions

In the modern digital economy, data has become one of the most valuable assets for organizations. From customer interactions and…

Apr 22, 2026 · 4 min read · 0 claps #data #data-lake #data-lake-solutions

TCH

SY Syed M

How I Cracked the Databricks Certified Data Engineer Associate Exam in Just One Week

A practical, no-fluff guide from a seasoned software engineer who went from zero Databricks knowledge to certified in seven days.

Apr 21, 2026 · 5 min read · 7 claps #databricks #data-lake #spark

AI ECO TCH MKT

RO Rotimi Ademola

Stop Building Data Lakes. Start Building Data Products.

The data lake holds fifty petabytes. The CEO asks for a customer churn prediction. The team quotes six weeks. This is not a technology…

Apr 18, 2026 · 5 min read · 5 claps #data-lake #product-data #data-value

ECO MDA

GR Graylog · The Visibility Layer by Graylog

Your SIEM Charges for Logs You Never Search

Jeff Darrington, Technical Marketing Director, Graylog

Apr 17, 2026 · 10 min read · 0 claps #siem #data-lake #data-routing

TCH

AH Ahmed M Khan

When Your Data Hub Becomes a Bottleneck: A Practical Path to Data Mesh

For many organizations, the data hub or data lake started as a powerful idea. Centralize all data, build shared pipelines, and enable…

Apr 11, 2026 · 5 min read · 0 claps #data-mesh #data-mesh-implementation #data-lake

ART TCH

PH Phillip Moxley · Towards Data Engineering

3 Signs Your Business Has Outgrown Its Data Lake Architecture

And what to do before the cracks become craters

Apr 03, 2026 · 9 min read · 51 claps #data-lake #data-lake-consulting #data-lakehouse

ECO MDA

GR Graylog · The Visibility Layer by Graylog

Why Your SIEM Ingest Costs Are Too High (And How to Fix It)

Jeff Darrington, Technical Marketing Director, Graylog

Mar 28, 2026 · 6 min read · 0 claps #siem #data-lake #log-management

TCH

SH Shivam Rawat

The Hidden Data Gaps Holding Your Business Back — And the Role of Data Lake Consulting

Most businesses today are not struggling because they lack data. They are struggling because their data is incomplete, scattered…

Mar 23, 2026 · 6 min read · 20 claps #data-lake-consulting #data-lake #data-lakehouse

TCH LIF MKT

SH Shivam Rawat

Q1 2026 Report: Key Trends in Data Engineering & AI Adoption

The first quarter of 2026 highlighted a major shift in how organizations approach data and analytics. Businesses are moving away from…

Mar 09, 2026 · 4 min read · 51 claps #data-trend #data-engineering #data-warehouse

TCH

MU Murat Demirbaş

Veri Ambarı vs Veri Gölü: Kahve Makinesi mi, Çuval Kahve mi?

Sabah kahvesini düşünelim. Çoğumuz kahve makinesine hazır kapsülü koyar, düğmeye basar ve birkaç dakika içinde içmeye hazır kahvemizi…

Feb 10, 2026 · 5 min read · 0 claps #data-lake #data-warehouse #etl