Building Generative AI Capabilities with Databricks as a Foundational AI Platform

Databricks launched **Lakehouse AI** in 2023 as a unified capability encompassing Data warehouse, Lakehouse, Machine learning, and Generative AI as depicted below. This article briefly introduces Databricks AI capabilities and shares early feedback/observations based on experimentation with their platform, particularly for Generative AI.

Image Source: https://www.databricks.com/blog/lakehouse-ai

Since it acquired MosaicML in 2023, Databricks has expanded its data engineering, machine learning, and generative AI capabilities as an end-to-end solution with key offerings as:

• **DatabricksIQ**: a Data Intelligence Engine providing a natural language interface across the entire Databricks environment including Unity Catalog, dashboards, notebooks, data pipelines, and documentation.

Source: Databricks

• **Mosaic AI Gateway**: a centralized service that governs, monitors, and provides production-level readiness with usage & security guardrails for accessing generative AI models. It is integrated with the Databricks ecosystem such as Unity Catalog for governance.
• **Mosaic AI Model Serving**: provides a unified interface to deploy, govern, and query AI models including LLMs. Opensource foundation models such as Llama 3.1/3/2, Mistral, and more are available for immediate use with pay-per-token pricing. External models (not hosted with Databricks such as GPT-4) can be governed using Mosaic AI.
• **Databricks Assistant**: An embedded AI assistant with the Databricks platform that provides context-aware suggestions for notebooks, SQLs, jobs, and more.
• **Gen AI Playground**: Provides a chat-like interface to test prompts and compare LLMs available through the Databricks platform:

Key Observations and Feedback on Databricks AI Platform

Considering my corresponding knowledge on similar platforms such as Google Cloud Vertex AI, Microsoft Azure AI, Amazon Bedrock, Snowflake Cortex, and more — the views and observations represented below are based on my personal experience only categorized across broader areas.

Lakehouse and Generative AI Capabilities Integration

Strengths

• With DatabricksIQ as an AI-based intelligence engine, most of the Databricks functionalities across notebooks, unity catalog, data pipelines, and more have embedded Generative AI capabilities providing natural language conversational skills. For example — it recommends AI-generated column descriptions based on the table’s metadata and structure or creating a widget on the dashboard using natural language:

Databricks AI-assisted Developer Experience

• Intelligent search across the platform, self-optimization capabilities, AI-powered governance with unity catalog, and more elevate the overall Lakehouse architecture.
• Data enrichment capabilities of the Databricks platform have been well-established and empowered by the Unity catalog. Open sourcing unity catalog further strengthens Databricks brand as a transparent and community-friendly company.

Areas of Improvements

• More transparency around the DatabricksIQ on how it uses organizational data (e.g. naming conventions, standards, processes, etc.) to elevate the GenAI capabilities, particularly for the GenAI assistant.
• No code or low-code experience such as Agent builder functionality available with Google Vertex AI or Azure AI is yet to be available as part of the Databricks platform. The integrated application development experience is limited and while the data engineering aspects of the platform has been appreciated, the application and business user interface can be improved further.

Vector Search Capabilities

Strengths

• Vector search is available as part of the Databricks platform as a fully managed compute capability with features such as creating vector embeddings automatically from files in the Unity Catalog.

Areas of Improvements

• Provisioning of vector search endpoints can be a time-consuming activity. Additionally, vector index creation with embedding generation takes longer as compared to other industry-leading Vector databases based on my experience (the comparative benchmark metrics need to be analyzed & published further).

AI Gateway Capabilities

Strengths

• Databricks contributed MLflow to the Linux Foundation as an open source project. As part of the integrated offering — MLflow AI Gateway — provides gateway capabilities to manage credentials for SaaS models or model APIs and provide access-controlled routes for querying.
• Mosaic AI Gateway is part of the broader AI gateway considering MosaicML acquisition and being integrated as part of the platform.

Areas of Improvements

• A limited set of guidelines are available as MLflow API Gateway is being deprecated in favor of Mosaic AI Gateway. Databricks Assistant provided instructions but the Mosaic AI gateway is not easily accessible.

Foundation Model Access

Strengths

• Basic AI Playground (see below) is available to experiment with the foundation model and it offers a curated list of 80+ models available within Databricks Marketplace — including MPT-7B, Falcon-7B, Llama 3.1, DBRX, AI21 Jamba 1.5, and more.
• Databricks focuses on providing custom models and fine-tuned models for enterprises in comparison to using managed closed-source models such as OpenAI GPT4 or Google Gemini.

Source: Databricks AI Models Marketplace

Image: Snapshot of AI Playground from Databricks UI (along with Databricks Assistant)

Areas of Improvement

• In comparison to Google Vertex AI or Azure AI or Amazon Bedrock — Databricks needs to expand its partnership to provide additional foundation models. Comparatively, Google Vertex AI and Azure AI have more than models listed on their respective portals.

To conclude, Databricks is emerging as a formidable Generative AI platform, particularly with a unified offering as Lakehouse and MosaicAI integration. Considering the competitive landscape, the innovation and offerings in Generative AI capabilities will continue to evolve with companies like Databricks, which are opensource-friendly.

Disclaimer:

All data and information provided on this blog are for informational purposes only. The author makes no representations as to the accuracy, completeness, correctness, suitability, or validity of any information on this blog and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. This is a personal view and the opinions expressed here represent my own and not those of my employer or any other organization.