Apache Spark WTF??? 😈 Vices & Virtues 😇

Welcome, brothers and sisters of the bit, to the cathedral of Spark Data Sources V2.

DSv1 is the little demon on your shoulder whispering: “Just read the folder.” DSv2 is the angel with better contracts: cleaner scan planning, richer catalog integration, column pruning, and pushdown when the connector actually supports it.

Because in Spark, salvation is not promised. It’s pushed down.

The internal fight of every Spark engineer: the good, modern, not-yet-fully-known DSv2 path versus the old, bumpy, daemon-haunted DSv1 road. One promises cleaner contracts. The other promises familiarity.

“No te vamos a enseñar nada que no sepas Ni nada que no quieras aprender” (adapted from **“Vicios y Virtudes” by Doble V**)

🤥 The Pretty Tiny Lie

The prettiest lie in data engineering is this: spark.read.parquet("/path").

It looks like Spark politely opens a file, adjusts its tiny reading glasses, and starts reading. I’m afraid is not that simple.

Spark is a distributed engine with trust issues. A DataFrame is not “data loaded into memory.” It’s a lazy promise. Nothing truly happens until an action appears and says: Now let’s spend some money.

That laziness matters. It gives Spark time to prune columns, push filters, and avoid turning your cloud bill into a medieval punishment device.

Enter the V1 Demon

Then comes the old demon: Data Source V1. DSv1 was simple, useful, and very 2010s.

Spark asked the source: Can you give me rows? The source answered: Sure, boss.

Under the hood, connectors usually exposed a BaseRelation, then chose their level of repentance.

The Three Confession Modes

• **TableScan** read everything and confessed later.
• **PrunedScan** at least skipped unnecessary columns.
• **PrunedFilteredScan** pushed some filters too, although Spark could still re-check them afterward because production has trust issues for a reason.

Where V1 Started Sweating

This worked fine when “data source” mostly meant “some files in a path.”

But modern lakehouse tables are not just folders with ambition. They have catalogs, snapshots, partition rules, write semantics, deletes, updates, and users who say “quick overwrite” five minutes before lunch.

That is where V1 started sweating.

Its contract was too vague. Capability negotiation was clumsy. Catalog awareness was weak. Writes were stretched beyond their original comfort zone. And every serious connector had to hide too much cleverness behind too little API.

🎬 Second Parts Were Never Good… Really?

There’s a dangerous phrase in technology: We are building V2. Every engineer in the room immediately ages seven years.

Software sequels have a bad reputation. Jaws 2. Speed 2. Your company’s second attempt at a “unified data platform.” All of them introduce new villains, worse lore, and at least one interface named like a tax form.

So when Spark brought Data Source V2 into the story, suspicion was healthy. But this sequel was necessary.

The world had changed. Connectors were no longer just answering: “Can you give me rows?” They were being asked about catalogs, schemas, partitions, pushdowns, columnar reads, batch writes, streaming, table capabilities, and whether they could please stop behaving like a folder with anxiety.

V1 was not bad. It was underdressed. It arrived at the lakehouse wedding wearing flip-flops.

The Three Curses of V2

DSv2 first appeared experimentally in Spark 2.3.0 and has evolved ever since. It had to survive three curses.

• Curse one: fix the old pain. V1 made too many things implicit: what could be pushed down, what could be pruned, what a table could do, and how writes should behave.
• Curse two: don’t break the kingdom. Spark couldn’t just delete V1 and tell half the planet: Good luck, sinners. Too many production jobs, connectors, and legacy configs depended on it.
• Curse three: predict the lakehouse future. Tables stopped being innocent paths. They became catalogs, snapshots, transactions, partition specs, governance rules, and angry business expectations wearing a hoodie.

The Job Interview with Catalyst

DSv2 changes the conversation between Spark and external systems. Instead of one vague connector contract, DSv2 uses smaller interfaces under org.apache.spark.sql.connector.

A connector can explicitly say what it supports: batch reads, columnar reads, filter pushdown, aggregate pushdown, limit pushdown, catalog operations, and table capabilities.

That matters because Spark cannot optimize what the connector cannot express. If the connector says, “Just give me everything, I’ll filter it later,” Catalyst has limited room for miracles.

Writes, Batches, and Adult Supervision

DSv2 gives Spark a more structured write path, with writer factories, commit messages, driver-side commits, and abort hooks. That does not magically make every connector transactional, but it gives serious table formats a cleaner protocol to implement serious behavior.

The same idea applies to performance. DSv2 lets connectors opt into columnar reads through ColumnarBatch, instead of always pretending the universe is made of individual rows. When the source supports it, Spark can stay closer to its vectorized execution path and avoid unnecessary object-shaped suffering.

Add catalog support, and Spark graduates from “read this suspicious path” to “operate on this named table with capabilities.”

🚀 Spark 4.x: New Capabilities and New Demons

By the time we reach the Spark 4.x era, DSv2 is no longer an experimental preview — it’s the unchallenged sovereign of the execution engine. But with supreme power comes supreme temptation, and Spark 4.0 unleashes a whole new legion of migration demons to test your faith.

Python Enters the Connector Cathedral

Spark 4.0 made custom PySpark sources possible. Spark 4.1 made them sharper with filter pushdown: Python connectors can receive predicates earlier and read less data before rows cross into Spark.

*😇 *The virtue: less data movement, less Python/JVM boundary pain, fewer “why did this API reader fetch the entire internet?” moments.

*😈 *The vice: pushdown is a promise. If your Python source mishandles dates, nulls, decimals, or time zones, it will be wrong faster — which is still wrong.

Native XML and Framework Hardening

Spark 4.0 expands its holy catalog by promoting XML to an official, built-in data source, exorcising the brittle external package dependencies of yesteryear. Simultaneously, the broader DSv2 framework receives massive engine-level hardening:

• Storage Partitioned Joins: Eliminates expensive network shuffles by directly exploiting a V2 source’s native data distribution.
• Aggregate & Percentile Pushdown: Allows advanced source collaboration, executing intensive functions like percentile calculations directly on the storage layer before bytes ever hit the network.
• Catalog Procedures: Introduces explicit stored procedure catalog APIs and native support for structural metadata queries.

The Migration Goblins: Schema and Default Shifts

Spark 4.x brings better optimization, but also sharper migration traps.

JDBC got more transparent — and stricter. Spark 4.0 can show the raw SQL sent to the remote database in EXPLAIN, which is great for debugging why your database is crying. But JDBC type mappings also became stricter, especially around smaller numeric types and floats. Loose downstream schemas may suddenly start failing like they found religion.

ANSI mode is now the default. Bad casts, overflows, and parsing errors are less likely to become silent nulls. Good for correctness, painful for legacy pipelines. And if a pushed expression behaves differently in the external system than in Spark congratulations: you optimized your way into wrong data.

S3 commit behavior changed in Spark 4.1. The Hadoop S3A Magic Committer is now enabled by default for S3 buckets. That can reduce painful rename-based write costs, but commit mechanics are not a casual setting. Test your table formats, retries, aborts, and failure paths unless you enjoy orphan files with emotional depth.

🛠️ The Virtue Map: What DSv2 Wants to Fix

By now, the demon has confessed: DSv1 was useful, battle-tested, and too vague for the lakehouse era.

DSv2 does not fix everything. This is not a spa weekend for cursed connectors. It mainly fixes the conversation between Spark and external systems: what can be pushed, pruned, written, trusted, and treated as a real table.

Because most distributed data disasters do not start with evil. They start with ambiguity.

Explicit Capabilities: No More Guessing Games

DSv1 often felt like reading a connector’s personality through horoscope traits.

DSv2 is more direct. A source can explicitly expose what it supports: reads, writes, catalog operations, filter pushdown, aggregate pushdown, limit pushdown, columnar reads, partition reporting, and other useful sins.

The virtue is obvious: Spark can plan with more information.

*😇 *The virtue: better conversations with Catalyst.

*😈 *The vice: a capability is a promise. If a connector says, “Yes, I pushed that filter,” but actually evaluates nulls, time zones, decimals, or strings differently from Spark, congratulations. You did not optimize the query. You optimized the truth out of the data.

Table-Oriented Writes: From Paths to Actual Tables

Old Spark writes were often path-shaped chaos: Save this DataFrame over there.

🤔 Okay, but… what exactly is “there”? A directory? A table? A partition? A business-critical asset?

DSv2 pushes Spark writes through DataFrameWriterV2 and catalog-aware APIs. Instead of blindly dropping files into paths, Spark can target named tables and use clearer operations like append, create, replace, conditional overwrite, or dynamic partition overwrite.

*😇 *The virtue: it makes intent more explicit. But explicit doesn’t mean invincible. The connector still has to implement distributed writes correctly. Task failures, driver failures, retries, partial files, commit messages, and abort logic are still where optimism goes to open a Jira ticket.

*😈 *The vice: DSv2 gives better write machinery. It does not eliminate the need to test the machinery.

Columnar Execution: Less Row-Shaped Suffering

DSv2 lets connectors participate in columnar reads, using ColumnarBatch data when they support it. That can keep Spark closer to its vectorized execution path and reduce unnecessary object-shaped suffering.

*😇 *The virtue: less pointless row conversion. Better CPU behavior. Cleaner integration with modern columnar formats.

*😈 *The vice: columnar paths are less forgiving. Memory lifecycle bugs, off-heap pressure, oversized batches, and badly managed readers can turn performance engineering into a haunted escape room.

The Planning Chain: Who Does What?

DSv2 also makes the connector lifecycle more structured.

The rough path looks like this:

TableProvider
 ⤷ Table
    ⤷ ScanBuilder
       ⤷ Scan
          ⤷ Batch
             ⤷ InputPartition
                ⤷ PartitionReaderFactory
                   ⤷ PartitionReader

The driver plans the work. Executors do the work.

That separation matters. The driver should build the plan, push down what can be pushed, and create lightweight partition descriptions. Executors should create the actual readers and touch the external system.

*😇 *The virtue: cleaner separation between planning and execution.

*😈 *The vice: put a heavy client, open socket, or non-serializable monster on the wrong side of that boundary and Spark will remind you that distributed systems have feelings. Mostly negative ones.

🏢 Data Source V2 Architecture: The Angelic Bureaucracy

DSv2 is Spark’s angelic bureaucracy: many small interfaces, each with a job, a clipboard, and a deep fear of ambiguity.

The point is not elegance for elegance’s sake. The point is survival.

Spark is lazy, distributed, optimized, failure-prone, and suspicious. DSv2 separates planning from execution so the Driver can decide what should happen, while Executors handle the messy business of touching data.

*👑 *Golden rule: The Driver plans. Executors read. Mix them up and the demon gets root access.

The Driver-Side Chain

On the Driver, Spark negotiates the plan before real distributed reading begins.

**TableProvider** is the receptionist. It maps user options and format strings into a source Spark can understand.

😇 The virtue: clean entry point.

😈 The vice: do not turn the receptionist into a forklift. Heavy listings, network calls, or expensive inference here can slow the whole job before it even starts.

**Table** is the logical asset: a table, stream, directory-backed source, or external system. It exposes schema and capabilities.

😇 The virtue: Spark can ask what the source claims to support.

*😈 *The vice: capabilities are testimony. Lying here turns planning into fan fiction.

**ScanBuilder and `Scan`** are the negotiation room. This is where Spark may push filters, prune columns, apply limits, or ask for other optimizations before the scan is frozen.

😇 The virtue: less useless data enters the engine.

*😈 *The vice: pushdown is only safe when source semantics match Spark’s expectations. Nulls, time zones, decimals, and strings are where optimism goes to get audited.

**Batch** is the project manager. It turns the scan into parallel work through planInputPartitions() and provides the reader factory.

😇 The virtue: the source controls how work is split.

*😈 *The vice: one giant partition turns the cluster into one sad laptop. Millions of tiny partitions turn the Driver into an air-traffic controller on espresso.

The Executor-Side Chain

After planning, Spark ships lightweight work descriptions to Executors.

**InputPartition* is the work ticket. It should describe what slice to read*, not contain the data itself.

😇 The virtue: tiny, serializable, task-specific.

*😈 *The vice: put open sockets, live clients, or non-serializable beasts inside it and Spark will throw the chair.

**PartitionReaderFactory** is the toolkit. Spark serializes it to Executors so tasks can create readers locally.

😇 The virtue: expensive physical setup can happen near the task.

*😈 *The vice: initialize heavyweight clients too early and you may accidentally smuggle driver-side state across the network like a raccoon in a suitcase.

**PartitionReader** is the worker. It runs inside a task, advances with next(), returns data with get(), and cleans up with close().

😇 The virtue: isolated, local, precise reading.

*😈 *The vice: retries happen. Readers must handle failure, cleanup, and side effects carefully, or your connector becomes a duplicate generator with a badge.

Rows, Batches, and Local Shortcuts

DSv2 can read data in different physical shapes.

Row-based reads emit InternalRow. Reliable, familiar, and sometimes perfectly fine.

Columnar reads can emit ColumnarBatch, letting Spark stay closer to vectorized execution when the connector supports it.

😇 The virtue: fewer row-shaped objects, better analytical throughput, less CPU suffering.

*😈 *The vice: columnar code is fast, sharp, and unforgiving. Batch lifecycle bugs and memory pressure can turn “high performance” into “why is the executor gone?”

Then there are local scans, where Spark can avoid distributed scheduling for tiny local results.

😇 The virtue: great for small metadata-like answers.

*😈 *The vice: never confuse “local” with “free.” If something unexpectedly grows, the Driver becomes the sacrifice.

Practical Smell Tests

Make the connector confess with df.explain("formatted").

• If a huge query creates one scan task, partition planning has collapsed. Your cluster is now decorative furniture.
• If a narrow query still reads the full schema, column pruning failed. You are paying the broad schema tax.
• If a filtered query shows no pushed predicates or useful pruning, Spark is doing cleanup work the source could have done earlier.
• If retries create duplicates outside Spark, your reader or writer has side effects it cannot safely repeat. That is not a retry problem. That is a correctness crime scene.

DSv2 architecture, explained as Spark’s holy paperwork department: the Driver plans, Executors read, and every connector must choose between angelic pushdown discipline or demonic “just scan everything” chaos.

📉 Pushdown: The Virtue That Saves Your Cluster

“Pushdown” is one of those words data engineers say with the confidence of medieval priests blessing a battlefield. Everyone nods.

Then a query scans 48 TB to return twelve rows, and suddenly the whole team becomes deeply spiritual about EXPLAIN FORMATTED.

The idea is simple: do work closer to the data. The dangerous part is this: pushdown delegates part of Spark’s query meaning to another system.

😇 The virtue: if both systems agree, you save I/O, CPU, memory, and possibly your monthly budget review.

*😈 *The vice: if they disagree, you get wrong answers at incredible speed.

The Optimization Conversation

Spark does not randomly throw wishes at a connector. With DSv2, Catalyst talks to the source through explicit interfaces. Depending on what the connector supports, Spark may try to push filters, prune columns, push aggregates, apply limits, report statistics, or use runtime filters.

But this is not a magic conveyor belt. The source must clearly say what it accepted, what it rejected, and what Spark still needs to evaluate afterward.

😇 A good connector says: “I handled this predicate. I could not safely handle that one. Please re-check this after the scan.”

😈 A bad connector says: “Trust me.”

And that is how optimization becomes fan fiction.

Pruning vs. Pushdown: Stop Mixing the Spells

Not every optimization is the same spell.

• Column pruning reduces width. Spark asks for only the columns needed by the query.
• Partition pruning skips entire partitions or files using metadata.
• Predicate pushdown sends filters closer to the source.
• Aggregate pushdown lets the source compute things like counts, mins, maxes, or partial aggregates.
• Limit and Top-N pushdown ask the source to stop early or return only the best matching rows.

Same family. Different demons. Calling everything “pushdown” is how meetings become cheaper and incidents become expensive.

Column Pruning: The Width Diet

Column pruning sounds boring. That is why it is beautiful.

If your query needs two columns from a 500-column table, Spark should not drag the whole furniture store into the executor just to use one chair.

With DSv2, a connector can implement required-column pushdown so the scan reads only the needed schema.

😇 The virtue: less I/O. Less decoding. Less network traffic. Less executor memory drama.

*😈 *The vice: nested schemas are where confidence goes to die. Top-level pruning is easy. Deep structs, case sensitivity, field ordering, schema evolution, and nested projections are the real exam. A connector that prunes customer_id but panics at payload.device.geo.country is not optimized. It’s cosplaying.

*💡*Audit hint: run df.explain("formatted") on a narrow query. If the read schema still looks like the whole table walked into the room, you are paying the broad schema tax.

Predicate Pushdown: Fast, Sharp, Suspicious

Predicate pushdown sends filters into the source so Spark reads less data.

For JDBC, this may become a WHERE clause. For files and lakehouse tables, it may use metadata, file statistics, partition values, or data skipping structures to avoid reading irrelevant chunks.

😇 The virtue: massive data reduction before Spark starts sweating.

*😈 *The vice: the source must mean the same thing Spark means. Nulls. Decimals. Time zones. String comparison. Case sensitivity. NaN. Timestamp boundaries. If Spark says amount > 100.00 and the source interprets precision or null logic differently, your query did not get faster. It got creatively wrong.

*👍 *The good news: DSv2 supports partial pushdown. A connector can accept the predicates it understands and leave the risky ones for Spark to re-check after the scan.

Aggregate, Limit, and Top-N Pushdown

Some pushdowns are simple filters with gym clothes. Others walk in wearing a PhD, a monocle, and a loaded footgun.

Aggregate Pushdown. Aggregate pushdown asks the source to compute summaries before Spark reads the raw rows.

Instead of dragging billions of records into Spark just to calculate count(*), min(amount), max(event_time), or some grouped summary, Spark can ask the source: Hey, do you already know the answer?

A table format may answer count(*) from metadata. A file index may know min/max statistics. A database may compute the aggregation far more efficiently than Spark reading everything over the network.

😇 The virtue: a massive scan becomes a tiny answer.

*😈 *The vice: Math has opinions. Decimals, floating point, nulls, NaN, grouping behavior, overflow handling, and partial aggregation semantics must match Spark’s expectations. If the source computes “average” or “count” with slightly different rules, you do not get acceleration. You get a fast lie with a PhD.

Limit Pushdown. Limit pushdown asks the source to stop early.

For previews, sampling-like exploration, API-backed sources, JDBC queries, or expensive scans, this can be extremely useful. If Spark only needs 100 rows, the connector may avoid reading 100 million.

😇 The virtue: Spark does less work, reads less data, and gets quick answers for bounded queries.

*😈 *The vice: a raw LIMIT without ordering is not a promise of stable rows. It’s Spark saying: “Give me some.” That may be fine for previews. It is not fine when somebody thinks those rows are deterministic, representative, or business-certified truth.

Limit pushdown saves time. It does not create meaning.

Top-N Pushdown. Top-N pushdown is limit pushdown after sorting enters the room.

Instead of asking for “some 10 rows,” Spark asks for “the best 10 rows according to this ordering.”

That can be powerful. A source with indexes, clustering, sorted files, or database execution can often find the top rows without scanning everything.

😇 The virtue: You avoid reading the whole dataset just to sort it and throw almost everything away.

*😈 *The vice: Ordering semantics must match exactly. Null ordering, string comparison, case sensitivity, collation, timestamp handling, and tie-breaking can all change the result. If Spark and the source disagree, your top 10 becomes: “Top 10 according to a different religion.”

Runtime Filters, Statistics, and Python Sources

Some pushdown decisions happen before execution. Some arrive fashionably late.

• Runtime filtering lets Spark infer filters during execution, often from joins, and use them to reduce already-planned V2 input partitions when beneficial.
• Statistics reporting lets a source estimate scan size and row count after pushdowns. Good stats help Spark choose better joins. Fantasy stats help Spark confidently choose disaster.
• Python Data Source filter pushdown arrives in Spark 4.1, extending the Python Data Source API introduced in Spark 4.0. That means Python-first custom sources can now participate in predicate pushdown and reduce data movement.

This is great. It also means your Python connector can now be wrong faster.

The Museum of Production Sins

Pushdown failures usually arrive wearing one of four cursed costumes.

• The Silent Tax. The connector pushes down nothing. Results are correct, but Spark reads the universe and your bill starts writing poetry.
• The Theater Kid. The plan claims pushdown happened, but the connector still reads everything. The UI looks optimized. The storage layer is crying.
• The Staging Angel, Production Goblin. Pushdown works in staging because stats, indexes, partitions, and case settings are perfect. In production, one drifted config turns the miracle into a full scan.
• The Fast Saboteur. The connector pushes something it does not truly understand: time zones, nulls, decimals, strings, NaN, or rounding. The query finishes instantly. The dashboard updates. The business celebrates. The data is haunted.

Pushdown: when your connector either saves the cluster… or returns haunted data at premium speed.

✍️ Writing with V2: The Demon with a Clipboard

Dear Storage Layer,

I hope this message finds you well, consistent, highly available, and not secretly full of orphan files.

I am writing to inform you that Spark would like to mutate a table. This time, we are using Data Source V2, so please prepare your catalog, your commit protocol, your partition logic, your cleanup hooks, and whatever ancient metadata altar you use to decide what is visible to readers.

So please, dear Storage Layer, stay healthy.

May your commits be atomic, your aborts be clean, your retries be idempotent, and your temporary files know when to disappear.

With cautious optimism, Spark

From Paths to Tables

DSv1 writing often felt path-shaped. DSv2 moves the mental model toward named tables through DataFrameWriterV2: df.writeTo("catalog.table")

Now the operation says what it means.

• .append() adds rows.
• .create() creates a table.
• .replace() replaces a table.
• .createOrReplace() does the obvious-but-still-scary thing.
• .overwrite(condition) replaces rows matching a filter.
• .overwritePartitions() dynamically replaces partitions touched by incoming data.

😇 The virtue: intent becomes explicit.

*😈 *The vice: explicit doesn’t mean safe. The connector still has to implement the behavior correctly, or your elegant API becomes a very well-documented crime scene.

Capabilities: The Table Testifies

A DSv2 table declares what it can do through TableCapability.

This is good because Spark can reject unsupported operations earlier instead of improvising with a flamethrower.

But capabilities are testimony. If a table says “I support overwrite by filter” and then implements it like “delete broadly, append nervously, hope aggressively,” Spark will build a plan around a lie. And distributed systems are extremely good at making lies expensive.

The Write Lifecycle

The DSv2 write path separates coordination from execution.

Table
 ⤷ WriteBuilder
    ⤷ Write
       ⤷ BatchWrite
          ⤷ DataWriterFactory
             ⤷ DataWriter
                ⤷ WriterCommitMessage

The Driver plans and coordinates. Executors write.

Each task gets a DataWriter, writes records, commits locally, and sends a small WriterCommitMessage back to the Driver.

If all tasks succeed, the Driver calls the global commit. If something fails, Spark calls abort.

That is the angelic version. The demonic footnote: Spark can retry failed tasks, but DSv2 does not magically make failed jobs transactional. The connector and table format still need solid commit logic, cleanup rules, and metadata guarantees.

DSv2 gives you the protocol. It does not give you immortality.

The Three Write Goblins

1. The Append Trap. Append sounds harmless. It is not. Retries, speculative execution, flaky networks, and external side effects can duplicate data if the writer is not idempotent.

If your DataWriter sends rows directly to a REST API mid-task and the task retries, congratulations: you built a duplicate cannon with enterprise branding.

2. The Overwrite Gap. overwrite(condition) is wonderfully clear. Replace only the rows matching this condition. But the table must make delete-plus-add atomic at the metadata layer.

If old data disappears before new data becomes visible, readers can fall into the data gap. That’s not an overwrite. That’s a trapdoor.

3. The Dynamic Partition Chainsaw. overwritePartitions() is great for daily reloads. It replaces only partitions touched by incoming rows. But it’s partition-level, not row-level reconciliation.

If the incoming data contains a cursed partition key, like dt = 2099-01-01, Spark will obediently create or replace the wrong kingdom.

Dynamic partition overwrite is useful. It’s also a chainsaw with calendar support.

Layout, Ordering, and Storage Reality

Some connectors can ask Spark to distribute or order data before writing.

😇 The virtue: better file layout today can make tomorrow’s reads faster.

*😈 *The vice: sorting and shuffling before a write is not free. You pay the tax now so future queries can enjoy the spa.

Object storage adds its own comedy.

On S3, commit protocols matter because rename-based file commits are historically painful. Spark 4.1 enables the Hadoop S3A Magic Committer by default for S3 buckets, which can improve write behavior, but it also means your failure-path tests are not optional.

Test driver failure. Test task retry. Test abort cleanup. Test the part nobody wants to demo. That is where the ghosts live.

🗂️ Catalogs: Where Paths Go to Become Institutions

A storage path tells you where bytes sleep. Cute, but useless for governance. It doesn’t know ownership, schema, namespaces, table properties or overwrite rules.

As platforms grow, paths become haunted addresses. Catalogs turn “some files over there” into named, governed, discoverable assets — slightly more likely to survive Monday.

From Path Strings to Named Assets

The path era was great for notebooks, experiments, and crimes committed before lunch. Modern teams need named assets: prod.sales.events.

That name carries meaning.

• prod is the catalog.
• sales is the namespace.
• events is the table.

Instead of guessing folder structure, Spark asks the configured catalog plugin to resolve the name.

spark.sql.catalog.prod = com.company.ProdCatalog

Two Doors Into DSv2

DSv2 has two important entry doors.

**TableProvider* is the tavern handshake. It’s used by V2 sources that don’t have a real catalog. It works well for format/options-based access: read this thing with these options.*

Useful, direct, and perfectly fine for many sources. But it cannot globally discover tables, manage namespaces, or handle metadata-changing operations like create and drop.

**CatalogPlugin** is the root interface for real catalog integrations. From there, Spark can layer table management, namespaces, functions, procedures, and richer metadata behavior.

😇 The virtue: names become first-class citizens.

*😈 *The vice: once names matter, misnaming things becomes a production incident with better stationery.

The Catalog Family Portrait

**TableCatalog** manages tables. It maps an identifier to a Table, and can support operations like create, alter, load, drop, or purge depending on the implementation.

😇 The virtue: Spark can treat tables as governed assets instead of mystery folders.

*😈 *The vice: methods like purge are not “oops-friendly.” Misconfigured lifecycle logic can turn routine cleanup into a data-loss speedrun.

**SupportsNamespaces** manages namespaces. Namespaces may behave like databases, schemas, tenants, projects, or whatever your platform architect saw in a dream.

😇 The virtue: you can organize assets cleanly.

*😈 *The vice: different catalogs interpret namespace rules differently. One team’s sales.events is another team’s “tenant folder with legal consequences.”

**StagingTableCatalog** supports staged table creation or replacement. This matters for CTAS and RTAS flows, where Spark should not expose a half-built table just because some tasks finished and the Driver is feeling optimistic.

😇 The virtue: metadata can become visible only after the data operation is ready to commit.

*😈 *The vice: staging is only as strong as the catalog implementation. If cleanup, abort, or visibility rules are weak, the stage becomes a trapdoor with branding.

Spark 4: More Verbs, More Paperwork

Spark 4.0 added catalog APIs for stored procedures, improved V2 table creation and CTAS support, supported ALTER NAMESPACE ... UNSET PROPERTIES, and added SHOW COLUMNS support for V2 tables.

That means catalogs are not just passive address books anymore. They are becoming operational control surfaces.

You can ask them to list things, alter things, create things, and in some systems call procedures that compact files, expire snapshots, repair metadata, or do other “please do not run this on prod by accident” activities.

😇 The virtue: more table operations become explicit and integrated with Spark SQL.

*😈 *The vice: every administrative verb is a loaded tool. CALL compact_table() is useful. CALL regret_everything() is usually called something else, but you will recognize it by the ticket volume.

Storage Partition Joins: When Catalogs Help the Optimizer

A good catalog does not only store names. It can help Spark understand layout.

When V2 sources report compatible partitioning and distribution information, Spark may avoid unnecessary shuffles in some joins.

😇 The virtue: less network chaos. More local work. Fewer executors reenacting a parcel delivery strike.

*😈 *The vice: layout claims must be true. If two tables say they are compatibly partitioned but reality disagrees, the optimizer may skip work that was actually necessary. And now your join is not faster. It is fiction with parallelism.

The Infernal Filing Cabinet

Catalogs solve problems. They also create more elegant problems.

The Caching Blindspot. Planning is faster when metadata is cached. But if an external system changes a table behind Spark’s back, stale metadata can turn a normal query into a surprise runtime failure.

The Ambiguity Glitch. Unqualified names are tiny grenades. A query that says events may resolve differently depending on session catalog, namespace, or environment. Today it hits prod. Tomorrow it hits a developer sandbox and returns suspiciously cheerful numbers.

The Trench Coat Table. This happens when the catalog says one thing and legacy jobs keep reading raw paths directly. The catalog table and the path table drift apart. Everyone thinks there is one dataset. There are actually two datasets in a trench coat.

🐍 Python Deep Dive: Data Sources Without the JVM Priesthood

For years, custom Spark sources lived behind the JVM curtain: Scala robes, Java incense, Maven at the temple door. Python users waited outside with a notebook whispering: I also have data.

Spark 4.0 opened the side entrance with the Python Data Source API. Not for rewriting transactional lakehouse engines during lunch, but for Python-first teams building internal API readers, synthetic generators, test fixtures, lightweight sinks, and “please just read this weird thing” connectors.

No JVM pilgrimage required.

The Anatomy of a Python Reader

A simplified reader flow looks like this:

DataSource
 ⤷ DataSourceReader
    ⤷ partitions()
       ⤷ read(partition)
          ⤷ Python iterator

**DataSource** is the entry point. It gives the source a name, declares a schema, and returns readers or writers.

**partitions()** creates the work tickets. These should be boring Python dictionaries or simple objects: paths, offsets, page numbers, IDs.

*😈 *The vice: never hide live clients, sockets, connection pools, or open file handles inside partition objects. Spark has to ship them to workers, and pickling a live network connection is how your job discovers interpretive dance.

**read(partition)** runs on the worker side. It receives one partition ticket and yields rows back to Spark.

*😈 *The vice: the yielded rows must match the declared schema. Field count, types, and nullability are not suggestions. They are the bouncer at the club. If your schema says integer and your API sends "lol", Spark will eventually find out.

Options and Schema Drift

Connector options arrive as strings. That means every option is innocent until parsed guilty.

😇 The virtue: a bad config dies at startup with a clear message.

*😈 *The vice: if you silently coerce everything into strings, your connector becomes a schema drift laundromat. External APIs change. JSON fields mutate.

Streaming, Pushdown, and the Spark 4.1 Upgrade

Spark 4.0 gave Python teams custom sources and sinks. Spark 4.1 makes the story sharper.

Python data sources can now participate in filter pushdown, meaning Spark can pass predicates to the source so it may read less data before rows cross into Spark.

😇 The virtue: less data movement. Less Python/JVM boundary pain.

*😈 *The vice: pushdown is a promise. If your Python source claims it handled event_time >= '2026-01-01', it must mean the same thing Spark means: nulls, time zones, strings, case sensitivity, decimals, all of it. Be conservative. Push simple predicates you truly understand. Return the rest to Spark for post-scan filtering.

Spark 4.1 also improves streaming Python data source support with Arrow writer support for streaming sources. That is useful progress, but do not translate “Arrow exists” into “my custom Python connector is now magically a zero-copy Formula 1 engine.”

The deepest circle of hell is reserved for benchmarks without context.

Writing Sinks and the Duplicate Goblin

Python sources can also write. That’s powerful. That’s also where the duplicate goblin lives.

If your Python sink sends row-by-row HTTP requests directly to an external REST API, one task retry can resend thousands of records while Spark reports a successful job. Congratulations. You built an invoice duplicator with a Spark logo.

*🛡️*Safer pattern: Use Spark to write to a durable staging area first. Then let a controlled delivery service handle retries, deduplication, rate limits, and external side effects.

Spark is great at distributed data processing. It’s not a therapist for flaky APIs.

Serialization Reality

Python data sources cross process and worker boundaries.

So keep state boring. Strings. Numbers. Small dictionaries. Simple configuration.

Don’t create database clients in constructors if they need to live on workers. Don’t pickle sockets and hope for mercy.

Initialize heavy resources inside the worker-side read or write logic. Wrap them in try...finally. Close them. The finally block is where engineering professionalism goes to stretch.

When to Use Python vs. JVM

Use the Python Data Source API when the team is Python-first, the integration is lightweight, the throughput is moderate, and the source is weird enough to need custom code but not important enough to summon a full JVM connector committee.

👍 Great fits: Synthetic data. Internal APIs. Testing mocks. Small custom sinks. Controlled ingestion helpers.

Use Scala or Java DSv2 when you are building serious infrastructure: high-throughput transactional table formats, complex catalog plugins, deep Catalyst integration, strict distributed commit protocols, or anything where “oops, duplicate data” could become a legal meeting.

⚖️ So, is DSv2 better than DSv1?

For modern data platforms, DSv2 wins. Not because V1 was stupid. V1 was built for a simpler world: “read these files and don’t ask too many questions.”

DSv2 lives in the lakehouse era: catalogs, pushdowns, table capabilities, columnar reads, distributed commits, and Python extensibility.

V1 is a garage. V2 is an airport. A garage is easier to understand. An airport is what you need when traffic, routing, security, failures, and angry passengers arrive together.

DSv2 did not invent the complexity. It made the demons sign the guest book.

So keep your filters pushed, schemas qualified, commits atomic, and flashlight on. In Spark, the next architecture ghost is never as far away as the roadmap says.

“Cuando Catalyst va a planearlo y tenéis un filtro, ¡pushdownearlo! No uso el V1: no hay razones para usarlo No tiro de paths: no hay razones para amarlo Para amarlo, co, ¡a pushdownearlo!” (adapted from **“Vicios y Virtudes” by Doble V**)

DSv1 fixes bikes with a rusty wrench. DSv2 manages global logistics with a halo, a clipboard, and trust issues. Same Spark, very different airport security. 😈😇