An Open Approach to Fine-Grained Access Control on External Engines

What a real enterprise modernization taught me about governance, federation, and why openness only works when it is auditable

By Ankur Kumar

Enterprise data is no longer confined to a single platform or cloud.

Most organizations now operate across a mix of storage systems, analytical engines, and AI tools — each introduced to solve a specific need. That flexibility has created enormous opportunity, but it has also raised the bar for governance. Fine-grained access control (FGAC) is no longer a secondary concern. It is foundational.

The challenge today is not simply connecting systems. It is ensuring that access remains consistent, auditable, and secure as data moves across platforms, engines, and organizational boundaries.

And that is where many federation strategies begin to strain.

As environments grow, permission models diverge, lineage fragments, and governance policies become harder to enforce consistently. Connectivity without governance quickly becomes operational risk.

Databricks approaches catalog federation with this reality in mind — grounding openness in a governance model designed to scale, remain predictable, and avoid unintended complexity at enterprise scale.

Enterprise data landscape with multiple engines and clouds unified through a central governance layer

Unity Catalog as the Governance Anchor

Unity Catalog is designed to act as the central control plane for data access and governance across the Databricks Lakehouse.

Instead of distributing governance logic across tools and integrations, the model is simple: keep governance centralized, even when compute and consumption are distributed.

That matters because in most enterprises, fragmentation does not happen all at once. It happens gradually:

• one team adopts a new analytics engine,
• another adds an external data store,
• a third introduces AI tooling,
• and suddenly the same dataset is governed differently depending on where it is accessed.

With Unity Catalog, the goal is to reverse that drift.

Access policies are defined once and applied consistently. Lineage and auditability are built in. Governance scales more cleanly as usage grows.

Unity Catalog also supports row filters and column masks, which are among Databricks’ strongest fine-grained access control capabilities. Row filters restrict which rows a user can see based on identity or group membership. Column masks dynamically redact or transform sensitive values at query time.

These controls are a major strength of Databricks-governed access within the platform, though it is important to note that some external access interfaces — including Iceberg REST Catalog and Unity REST APIs — do not currently support tables with row filters or column masks.

That distinction matters. Strong governance is not just about what the platform can enforce internally. It is also about understanding where the boundaries of that enforcement apply.

What a Real Enterprise Migration Revealed

The most important lesson I’ve seen in practice came from a global insurer modernization.

The environment had two parallel data platforms operating independently — a legacy data warehouse alongside a modern data lake platform. Each maintained its own access model, lineage process, and audit mechanisms.

On paper, both systems were functioning.

In practice, governance was fragmented. Permissions were defined differently across systems. Ownership was unclear. Visibility into data usage across platforms was limited. More than 8–10 business and analytics teams were affected, each compensating through local workarounds, manual controls, or duplicated datasets.

Consolidating governance into Unity Catalog became the turning point.

Permissions were standardized at the catalog, schema, table, and column level, replacing platform-specific rules with a single model. One of the biggest surprises during migration was how many legacy access patterns were implicit rather than documented — especially around shared service accounts. Those patterns had to be translated into explicit privileges before the model could be trusted.

That work was not trivial. But once it was done, lineage became transparent end to end, auditability improved materially, and governance incidents reduced.

Most importantly, governance stopped being a reactive control mechanism and became a proactive enabler.

That experience shaped how I now think about federation: the value of openness is not in how many systems can be connected. It is in whether those connections remain understandable and governable once the environment is under real operational pressure.

Openness Works Best When It Is Selective

There is sometimes a false trade-off in platform conversations: either support open ecosystem access, or preserve governance discipline.

In practice, the best enterprise architectures require both.

Databricks supports open ecosystem access patterns — including the Unity Catalog Iceberg REST Catalog API, which enables reading from and writing to supported Unity Catalog Iceberg tables from external Iceberg clients. That is an important investment in standards-based interoperability.

At the same time, Databricks takes a selective approach to federation into the platform — specifically, which third-party catalogs are federated as sources within the Databricks environment.

Generic federation vs certified integration, highlighting governance consistency

That selectivity is not about restricting openness. It is about preserving:

• governance clarity,
• operational predictability,
• and security controls at enterprise scale.

This is why Databricks emphasizes certified integrations — including MySQL, PostgreSQL, SQL Server, Amazon Redshift, Google BigQuery, Snowflake, and others — that behave predictably under enterprise conditions.

The principle is straightforward:

The goal is not openness at any cost. The goal is openness paired with reliable administration and auditable access.

Why Write Federation Needs More Caution

Read federation supports insight.

Write federation changes state — and that distinction matters.

Broad cross-catalog write federation introduces complexity that is difficult to reason about: distributed commits, long-running transactions, and partial failure scenarios that complicate recovery.

Rather than positioning Lakehouse Federation as a broad write-federation layer, Databricks emphasizes governed access patterns that preserve clear system boundaries and operational predictability. In most federation scenarios, the focus is read access with centralized governance through Unity Catalog.

Where external write patterns are required, they are best understood in the context of specific supported external access mechanisms or integrations with well-defined semantics — not as general cross-catalog write federation.

That preference is not just philosophical. It favors:

• transactional integrity,
• predictable rollback behavior,
• and simpler operational ownership.

Cross-catalog write federation vs. controlled write paths

Identity Matters Just as Much as Access

Passing user identities directly into external platforms can sound attractive in theory. In practice, it often introduces trade-offs around performance, metadata responsiveness, and tighter coupling to external identity providers.

There is also a broader security consideration around the expanded surface area where identity tokens must be protected and managed.

It is worth noting that Databricks supports governed storage access through Unity Catalog storage credentials and cloud-native identity mechanisms — including instance profiles, service principals, and managed identities, depending on cloud and configuration. That allows organizations to leverage existing identity infrastructure where appropriate.

Centralized identity management vs. identity passthrough

The key design choice is not to eliminate federation entirely. It is to keep policy resolution centralized within Unity Catalog, so organizations can optimize for both security and performance while maintaining fine-grained access control across catalogs, schemas, tables, and columns.

A Clear Industry Pattern Is Emerging

Across mature data platforms, a consistent pattern is emerging:

• support for open formats such as Apache Iceberg
• preference for certified integrations over generic connectors
• and a general avoidance of broad write federation in favor of controlled, well-defined access patterns

Databricks aligns with that direction, differentiating through the integration of these principles into a single governance model anchored in Unity Catalog.

That is ultimately the real differentiator: not openness alone, but openness that remains administratively reliable as the platform grows.

Final Thought

In enterprise environments, the goal is not maximum connectivity.

It is trusted access at scale.

Databricks’ approach to fine-grained access control and catalog federation reflects that reality. By anchoring governance in Unity Catalog and emphasizing certified, intentional integrations, organizations can adopt open data architectures without sacrificing reliability, security, or control.

The most important lesson from the field is that governance quality is rarely tested in architecture diagrams. It is tested during change.

It gets tested during migrations. It gets tested when teams rationalize access. It gets tested when someone needs to explain lineage across systems under delivery pressure.

That is where fragmented governance breaks down — and where a unified model proves its worth.

As data estates continue to evolve, the ability to balance openness with strong governance will define which architectures scale — and which break under complexity.

References

*Completing the Lakehouse Vision: Open Storage, Open Access, Unified Governance*

*AWS Glue Catalog Federation with Databricks Unity Catalog*

*Microsoft Fabric metadata mirroring and federation documentation*

*Google BigQuery external data sources documentation*