Databricks pushes generative AI with LakehouseIQ and AI tools

As the demand for generative AI continues to grow, Databricks is doing everything possible to put the technology at the heart of its data lakehouse.

Today, at its annual conference, the data and AI company announced LakehouseIQ, a generative AI tool democratizing access to data insights. Databricks also announced new Lakehouse AI innovations aimed at making it easier for its customers to build and govern their own LLMs on the lakehouse. 

The move follows the company’s $1.3 billion acquisition of MosaicML and comes at a time when Snowflake — Databricks’ main competitor — continues to make its own generative AI push. 

Databricks’ LakehouseIQ: An AI knowledge engine to query data

Most enterprise users today want to analyze data but are held back by a lack of technical expertise. For every analytical need, they have to go to data scientists and programmers, who then find and query the relevant datasets — a task that takes time and adds to the workload of already overworked teams.

With the addition of LakehouseIQ, Databricks is addressing this problem by offering a generative AI “knowledge engine” that allows anyone in an organization to search, understand and query internal corporate data by simply asking questions in plain English. No Python, SQL or data querying skills are needed. 

The offering uses elements like schemas, documents, queries, popularity and lineage to learn a business’ unique language (from internal jargon and data usage patterns) and immediately answer users’ queries. This level of understanding allows the solution to more accurately interpret the intent of a question and even generate additional insights to work with.

Plus, since it is fully integrated with Unity Catalog (Databricks’ flagship solution for unified search and governance), there’s always adherence to internal security and governance rules.

“LakehouseIQ solves two of the biggest challenges that businesses face in using AI: getting employees the right data while staying compliant, and keeping data private when it should be. It alleviates [the burden on] time-strapped engineers, eases the burden of data management, and empowers employees to take advantage of the AI revolution without jeopardizing the company’s proprietary information,” Ali Ghodsi, cofounder and CEO of Databricks, said.

Notably, Dremio and Kinetica are also exploring similar conversational data querying capabilities. And Snowflake itself has acquired Neeva, expected to enhance its ability to offer intelligent and conversational search experiences to enterprises that use its platform to store, analyze and share data. The data cloud company has also launched Document AI, a conversational tool to extract insights from unstructured documents.

While LakheouseIQ puts generative AI to use within Databricks’ platform, Lakehouse AI helps enterprises build generative AI solutions on the platform for their own use cases. This digital toolbox is now being enhanced to cover the entire AI lifecycle, from data collection and preparation to model development and LLMOps to serving and monitoring.

Databricks said it is expanding Lakehouse AI with vector embedding search to improve generative AI responses; a curated collection of open-source models (including MosaicML’s MPT-7B) available in the marketplace; LLM-optimized model serving; MLflow 2.5, with capabilities such as AI gateway and prompt tools; and lakehouse monitoring for end-to-end visibility into the data pipelines driving the AI efforts. 

“We’ve reached an inflection point for organizations: leveraging AI is no longer aspirational — it is imperative for organizations to remain competitive. Databricks has been on a mission to democratize data and AI for more than a decade and we’re continuing to innovate as we make the lakehouse the best place for building, owning and securing generative AI models,” Ghodsi added.

>>Follow VentureBeat’s ongoing generative AI coverage<<

At the conference, Databricks also introduced Delta Lake 3.0 with compatibility for Apache Iceberg and Hudi and federation capabilities that enable organizations to create a highly scalable and performant data mesh architecture with unified governance.

Databricks’ Data and AI Summit runs June 26–29 in San Francisco.