Aporia and Databricks are collaborating to improve real-time monitoring of ML models

Machine learning (ML) observation platform Aporia today announced a strategic partnership with Databricks. According to the companies, the collaboration is designed to empower customers using Databricks’ Lakehouse platform, AI capabilities, and MLflow offerings by providing them with advanced monitoring capabilities for their ML models.

Organizations can now monitor their ML models in real time by deploying Aporia’s new ML observation platform directly on top of Databricks, eliminating the need to duplicate data from their Lakehouse or another data source.

In addition, the integration with Databricks streamlines the monitoring process, the companies said, enabling billions of predictions to be analyzed without data sampling, production code changes or hidden storage costs.

“This means monitoring billions of predictions, visualizing and explaining ML models in production becomes effortless,” Liran Hason, CEO of Aporia, told VentureBeat. “Aporia supports all use cases and model types, providing flexibility for ML teams to tailor the platform to their specific needs.”

Real-time monitoring, customization

The new offering enables monitoring for anomalies such as drift, bias, degradation and data integrity issues and activates live alerts to popular communication channels, ensuring timely notifications.

The platform also provides real-time customizable dashboards and metrics, enabling each ML stakeholder to prioritize their key areas of interest and translate data science metrics into tangible business impact.

This is critical in industries such as lending, hiring and healthcare, Hason said, fostering a fair and transparent landscape in automated decisions.

“Organizations could now manage all ML models under a single hub, regardless of deployment,” said Hason. “This centralization improves collaboration, facilitates communication and streamlines model management, promoting continuous learning and efficient team workflows.”

Streamline data monitoring with ML Observability

Organizations have traditionally faced challenges when monitoring large amounts of data, often requiring data duplication from their data lake to their monitoring platform. But, Hason said, this approach leads to inaccuracies, overlooked issues, drift, false-positive alerts and difficulties in ensuring fairness and bias.

The new integration with Databricks addresses these pain points by enabling organizations to quickly monitor all of their ML models on Databricks in minutes.

In addition, the integration maximizes the benefits of existing database investments, even for use cases that require processing large volumes of predictions, such as recommendation systems, query ranking models, fraud detection models, and demand forecasting models.

“There’s no need to duplicate data into a separate monitoring environment,” explains Hason. “This provides a single source of truth derived directly from your data lake, simplifying data management and accelerating insights into actions. The integration improves the effectiveness of ML model monitoring and provides flexibility and freedom for organizations to leverage their existing ML and data infrastructure.”

Numerous use cases

The company said the new ML observation platform will support many use cases, including improving recommendation systems through real-time performance monitoring.

Organizations can use Aporia to improve their search ranking algorithms, understand click-through rates, and improve search results. In addition, Aporia’s real-time monitoring helps detect and prevent fraudulent activity, strengthening security and promoting customer confidence.

In addition, the platform ensures accurate forecasting in supply chain management and retail by monitoring demand forecasting models, enabling teams to optimize their response to deviations from forecasted demand. The platform’s observation capabilities will also help financial institutions monitor credit risk models, ensuring accurate and unbiased credit assessments and identifying potential biases.

The Databricks Delta Connector establishes a connection between Aporia and an organization’s Databricks Delta Lake, linking training and inference datasets to Aporia, explains Hason.

The platform stands out in monitoring large-scale data by effortlessly processing billions of predictions without resorting to data sampling, Hason said. This allows for a comprehensive and accurate assessment of model performance, which is particularly beneficial for organizations struggling with significant data volumes.

“No critical insight goes unnoticed, which guarantees thorough monitoring,” he added.

Unleash the power of data for informed decision-making

Hason said the partnership will play a critical role in driving the wider adoption of observability in the AI ​​and ML landscape as it addresses existing demand and a deeper understanding and recognition of observability as a critical element in AI and promotes ML.

He said the combination of a robust observation platform and a scalable data platform makes an attractive proposition for organizations investing in AI and ML. The companies are developing a unified tool that improves observability at scale, enabling organizations to make informed decisions and optimize their AI initiatives.

“The collaboration is specifically designed to provide a centralized, end-to-end, cost-effective solution, enabling organizations to make confident data-driven decisions,” added Hason.

Organizations can monitor all production data in minutes, ensuring fast time to value. This accelerated implementation quickly unlocks the benefits of the investment.

“These new functionalities can save organizations valuable resources that would otherwise be spent troubleshooting and solving problems,” said Hason.