Join top executives in San Francisco on July 11-12 to hear how leaders are integrating and optimizing AI investments for success. Learn more
Data engineering startup Prophecy gives a new twist to creating data pipelines.
The California-based company, known for its low-code SQL tooling, today announced data copilot, a generative AI assistant that can create trusted data pipelines from natural language prompts and improve pipeline quality with greater test coverage.
The capability has the potential to make pipeline development a breeze and save data engineers valuable time for other more pressing tasks. Previously, we’ve seen gen AI used for other aspects of data workflows, such as querying and cataloging data.
Along with the tool, Prophecy announced a new platform to help companies build gen AI applications on top of their private data.
How does data copilot help?
Historically, building a data pipeline has revolved around writing complex SQL code. Data engineers had to put in a lot of time and effort to bring business users’ desired pipelines to life. Then players like Prophecy came in, offering low-code solutions (a drag-and-drop visual canvas) to simplify things.
Now, as the next step in this work, Prophecy has introduced the data copilot, which only requires someone to say what they want in natural language.
Once a command is given, the platform uses it to suggest a pipeline that brings all the data together for the desired report. The user can then preview the pipeline and accept it, or decline to come up with anything else.
“This will enable companies to be less of a data engineering bottleneck as business data users and others can serve themselves… Further, these data products will also be made more consistent and of higher quality as Prophecy Data Copilot suggests transformations and expressions. too,” Raj Bains, CEO and co-founder of Prophecy told VentureBeat.
To accomplish this, Bains explained that Prophecy creates a comprehensive knowledge graph of a company’s data models that includes technical metadata associated with tables and schemas, business metadata from data catalogs, and historical queries and code run on SQL, Spark, or Airflow. This graph is then fed into an advanced large language model that translates the natural language user query into a performant data pipeline. The system also learns and improves based on user feedback, he added.
Platform to build gen AI apps
In addition to the data copilot, Prophecy adds a platform to build gen AI solutions like chatbots on top of private company data.
The offer works in a two-step process. First, a data engineer populates a knowledge warehouse with unstructured text in internal messaging systems, documents, support tickets, and more (converted to vectors). Next, the engineer builds a streaming inference pipeline, i.e. a chatbot, supported by OpenAI.
“When a user asks a question, a vector lookup is performed to retrieve the internal documents most relevant to that question,” Bains explains. “Those documents provide crucial context for answering the question posed. This context plus the question itself form the prompt, which is sent to OpenAI via APIs. The response is then sent back to the user.”
Both new offerings are available to use starting today.
VentureBeat’s mission is to become a digital city plaza where tech decision makers can learn about transformative business technology and execute transactions. Discover our Briefings.
