Technology Prophecy's generative AI assistant ushers in a new era...

Prophecy’s generative AI assistant ushers in a new era of data pipeline automation


Join top executives in San Francisco on July 11-12 to hear how leaders are integrating and optimizing AI investments for success. Learn more

Data engineering startup Prophecy gives a new twist to creating data pipelines.

The California-based company, known for its low-code SQL tooling, today announced data copilot, a generative AI assistant that can create trusted data pipelines from natural language prompts and improve pipeline quality with greater test coverage.

The capability has the potential to make pipeline development a breeze and save data engineers valuable time for other more pressing tasks. Previously, we’ve seen gen AI used for other aspects of data workflows, such as querying and cataloging data.

Along with the tool, Prophecy announced a new platform to help companies build gen AI applications on top of their private data.


Transform 2023

Join us on July 11-12 in San Francisco, where top executives will talk about how they integrated and optimized AI investments for success and how they avoided common pitfalls.

register now

How does data copilot help?

Historically, building a data pipeline has revolved around writing complex SQL code. Data engineers had to put in a lot of time and effort to bring business users’ desired pipelines to life. Then players like Prophecy came in, offering low-code solutions (a drag-and-drop visual canvas) to simplify things.

Now, as the next step in this work, Prophecy has introduced the data copilot, which only requires someone to say what they want in natural language.

Once a command is given, the platform uses it to suggest a pipeline that brings all the data together for the desired report. The user can then preview the pipeline and accept it, or decline to come up with anything else.

“This will enable companies to be less of a data engineering bottleneck as business data users and others can serve themselves… Further, these data products will also be made more consistent and of higher quality as Prophecy Data Copilot suggests transformations and expressions. too,” Raj Bains, CEO and co-founder of Prophecy told VentureBeat.

To accomplish this, Bains explained that Prophecy creates a comprehensive knowledge graph of a company’s data models that includes technical metadata associated with tables and schemas, business metadata from data catalogs, and historical queries and code run on SQL, Spark, or Airflow. This graph is then fed into an advanced large language model that translates the natural language user query into a performant data pipeline. The system also learns and improves based on user feedback, he added.

Platform to build gen AI apps

In addition to the data copilot, Prophecy adds a platform to build gen AI solutions like chatbots on top of private company data.

The offer works in a two-step process. First, a data engineer populates a knowledge warehouse with unstructured text in internal messaging systems, documents, support tickets, and more (converted to vectors). Next, the engineer builds a streaming inference pipeline, i.e. a chatbot, supported by OpenAI.

“When a user asks a question, a vector lookup is performed to retrieve the internal documents most relevant to that question,” Bains explains. “Those documents provide crucial context for answering the question posed. This context plus the question itself form the prompt, which is sent to OpenAI via APIs. The response is then sent back to the user.”

Both new offerings are available to use starting today.

VentureBeat’s mission is to become a digital city plaza where tech decision makers can learn about transformative business technology and execute transactions. Discover our Briefings.

Shreya Christina
Shreya has been with for 3 years, writing copy for client websites, blog posts, EDMs and other mediums to engage readers and encourage action. By collaborating with clients, our SEO manager and the wider team, Shreya seeks to understand an audience before creating memorable, persuasive copy.

Latest news

Guide on Hand Painted Journal Diary Crafting and Business Insights for Artisans

In a world where digital technology often overshadows the beauty of traditional craftsmanship, the creation of handpainted journal diaries...

Essen Ladies Exotische Verführerinnen

Entdecke|Erkunde|Tauche ein in} Deutschlands top Escort-Community: Ob du nach einem stilvollen Date suchst oder einfach nur ein schnelles...

Nutte Reutlingen – Erlebe Abenteuer

Entdecke|Erkunde|Tauche ein in} Deutschlands größte Escort-Community: Ob du nach einem stilvollen Date suchst oder nur ein schnelles erotisches...

Modelle Huren Hamburg Staat – Sinnliche Verführung

Entdecke|Erkunde|Tauche ein in} Deutschlands top Escort-Community: Egal ob du nach einem stilvollen Date suchst oder nur ein schnelles...

Modelle Huren Freiburg Anziehende Damen

Entdecke|Erkunde|Tauche ein in} dem deutschen größte Escort-Community: Ob du nach einem stilvollen Date suchst oder einfach nur ein...

Nutte Reutlingen – Entdecke Verheißung

Entdecke|Erkunde|Tauche ein in} dem deutschen führende Escort-Community: Ob du nach einem stilvollen Date suchst oder einfach nur ein...

Must read

You might also likeRELATED
Recommended to you