Dremio bets on generative AI, adds new tools to accelerate data workflows

Dremio, the open data lakehouse vendor that combines the capabilities of a data lake and warehouse on a unified layer, is going all in on generative AI.

The company today announced two new gen AI capabilities for its platform: a text-to-SQL experience for conversational querying of data and an autonomous semantic layer to help with data cataloging and processing.

The offerings will simplify working with data for Dremio users, enabling them to explore, discover and analyze their data assets, quickly and easily. Similar efforts have also been made by other leading players in the data ecosystem (including Snowflake and Informatica), signaling the rise of AI-driven data handling.

How will the new features help?

The task of unlocking value from data has long been dependent on a number of manual and time-consuming steps. With the latest capabilities, Dremio is bringing generative AI to address some of these gaps.

For instance, with the new Text-to-SQL experience, instead of spending time writing complex SQL (structured query language) queries, users can simply put in natural language inputs to get insights from their data. The offering leverages a semantic understanding of metadata and data and automatically converts the plain language query into SQL, providing the desired results. 

Similarly, the autonomous semantic layer leverages generative AI to eliminate the hassle of manual data cataloging. It automatically learns the intricate details of users’ data and produces descriptions of datasets, columns, and relationships to establish taxonomies for easy discovery and exploration of data. 

According to Dremio, this layer also learns from users’ workloads and creates reflections (optimized materialization of source data or a query) to accelerate data processing.

“By integrating Generative AI capabilities into our platform, we are accelerating data workflows and eliminating much of the manual work involved in SQL development, data catalog creation and curation, and more,” Tomer Shiran, cofounder and CPO of Dremio, said.

“Generative AI will transform data engineering, data science and analytics over the coming years, and we are excited to provide our users with the industry’s most powerful tools to uncover the true potential of their data,” he added.

Vector database capabilities

In addition to the gen AI tooling, Dremio is also integrating vector database capabilities directly in its lakehouse, enabling companies to build AI-powered applications without creating additional data silos. 

With this feature, users will be able to add a column of type vector to store and search embeddings for various data elements.

For instance, if a user has a table of Amazon reviews, they will be able to store the embeddings that encode the meaning of each review alongside other attributes. Then, when required, they could use Dremio’s indexes and SQL functions to retrieve similar or related reviews based on their meanings.

The text-to-SQL experience is now available for Dremio users, while the autonomous semantic layer and vector database capabilities will be rolling out at a later stage.