How observability, designed for data teams, can unlock the promise of DataOps

It is no exaggeration these days to say that every company is a data company. And if they aren’t, they must be. That is why more and more organizations are investing in the modern data stack (think: Databricks and Snowflake, Amazon EMR, BigQuery, Dataproc).

However, these new technologies and the increasing business critical nature of their data initiatives pose significant challenges. Today’s data teams must not only deal with the massive amount of data ingested daily from a wide variety of sources, but they must also be able to manage and monitor the maze of thousands of interconnected and interdependent data applications.

The biggest challenge comes down to mastering the complexity of the intertwined systems we call the modern data stack. And as anyone who’s ever spent time in the data trenches knows, deciphering data app performance, getting cloud costs under control, and mitigating data quality issues is no small feat.

When something breaks in these Byzantine data pipelines, with no single source of truth to point back to, the finger pointing begins with data scientists blaming operations, operations blaming engineering, engineering blaming developers — and so on and so forth for eternity.

Is it the code? Insufficient infrastructure resources? A planning coordination problem? Without a single source of truth that everyone can rally behind, everyone uses their own tool and works in silos. And different tools give different answers – and it takes hours (even days) to untangle the threads to get to the heart of the problem.

Why modern data teams need a modern approach

Data teams today face many of the same challenges that software teams once faced: a fractured team working in silos, under the gun to keep up with the accelerated pace of delivering more, faster, without enough people, in an increasingly complex environment .

Software teams successfully tackled those obstacles through the discipline of DevOps. A big part of what enables DevOps teams to succeed is the observability provided by the next generation of Application Performance Management (APM). Software teams are able to accurately and efficiently diagnose the root cause of problems, collaborate from a single source of truth, and enable developers to address problems early – before the software goes into production – without having to throw problems over the fence throw to the Ops team.

So why are data teams struggling when software teams aren’t? They essentially use the same tools to solve essentially the same problem.

Because despite the generic similarities, observability for data teams is a completely different animal than observability for data teams.

Cost control is crucial

First, consider that data teams not only need to understand the performance and reliability of a data pipeline, but also grapple with the issue of data quality: how can they be sure that they are feeding their analytics engines with high-quality inputs? And as more workloads move to an assortment of public clouds, it’s also vital that teams understand their data pipelines through the lens of cost.

Unfortunately, data teams find it difficult to get the information they need. Different teams have different questions that they need answers to, and everyone is myopically focused on solving their specific piece of the puzzle, using their own specific tool of choice, and different tools yield different answers.

Solving problems is a challenge. The problem can be anywhere in a very complex and interconnected application/pipeline for a thousand reasons. And while web app monitoring tools serve their purpose, they were never intended to absorb and correlate the performance details hidden in the components of a modern data stack, or “untangle the threads” between the upstream or downstream dependencies of a data application.

In addition, as more data workloads migrate to the cloud, the costs of running data pipelines can quickly spiral out of control. An organization with more than 100,000 data jobs in the cloud must make countless decisions about where, when, and how to perform those jobs. And every decision comes with a price tag.

With organizations relinquishing centralized control over infrastructure, it is essential for both data engineers and FinOps to understand where the money is going and identify opportunities to reduce/control costs.

Much observability is hidden in plain sight

To get granular insight into performance, costs and data quality, data teams are forced to pull together information from different tools. And as organizations scale their data stacks, the sheer volume of information (and resources) makes it extremely difficult to oversee the entire data forest when you’re in the trees.

Most of the detailed details needed are available – unfortunately they are often hidden in plain sight. Each tool provides some of the required information, but not all. What is needed is observability that brings all these details together and presents them in a context that makes sense and speaks the language of data teams.

Observability designed specifically for data teams from the ground up allows them to see how everything fits together holistically. And while there are a slew of cloud vendor-specific, open-source, and proprietary data observation tools that provide details about one layer or system separately, a full-stack observation solution can tie it all together into a workload. conscious context. Solutions using deep AI can further show not only where and why a problem exists, but also how it affects other data pipelines – and, finally, what to do about it.

Just as DevOps observability is the foundation to help improve the speed and reliability of the software development lifecycle, DataOps observability can do the same for the data application/pipeline lifecycle. But – and this is a big one but — The observability of DataOps as a technology should be designed from the ground up to meet the different needs of data teams.

DataOps’ observability spans multiple domains:

• Observability of data application/pipeline/model ensures data analytics applications/pipelines always run on time and error-free.
• Operational observability enables data teams to understand how the entire platform runs end-to-end, providing a unified view of how everything works together, both horizontally and vertically.
• Business perceptibility consists of two parts: profit and costs. The first deals with ROI and monitors and correlates data application performance with business outcomes. The second part is FinOps observabilitywhere organizations use real-time data to monitor and control their cloud costs, understand where the money is going, set budget protections, and identify opportunities to optimize the environment to reduce costs.
• Observability of data looks at the datasets themselves and performs quality checks to ensure correct results. It tracks origin, usage, and the integrity and quality of data.

Data teams cannot focus separately because issues in the modern data stack are interrelated. Without a unified view of the entire datasphere, the promise of DataOps will not be fulfilled.

Observability for the modern data stack

Extracting, correlating, and analyzing everything at a base layer in a data team-centric, workload-aware context yields five capabilities that are the hallmark of a mature DataOps observation function:

• End-to-end visibility correlates telemetry data and metadata from the entire data stack to provide a unified, deep understanding of the behavior, performance, cost, and health of your data and data workflows.
• Ambient awareness puts this aggregated information in a meaningful context.
• Actionable intelligence tells you not only what is happening, but also why. Next-generation observing platforms go a step further, providing prescriptive, AI-driven recommendations on what to do next.
• Everything happens through or makes a high degree of automation.
• This proactive ability is management in action, with the system applying the recommendations automatically — no human intervention is required.

As more and more innovative technologies find their way into the modern data stack – and more and more workloads migrate to the cloud – it becomes increasingly necessary to have a unified DataOps observable platform with the flexibility to understand the growing complexity and the intelligence to to provide a solution. That’s where DataOps observability.

Chris Santiago is VP Solutions Engineering for Untangle.