Report: Data engineers are extinguishing poor data quality 2 days a week

Past industry surveys have shown that most data professionals struggle with poor data quality despite the amount of time and resources invested in resolving it. To extend these insights and help data leaders understand where and how to invest resources, the data trust company, Monte Carlopartnered with Wakefield Research to survey more than 300 data professionals to find out the state of data quality in the broader market.

Results of the survey show that teams still struggle with data quality and waste valuable time, money and resources solving problems. Respondents indicated that an average of 40% of their time is spent evaluating or checking data quality. In addition, more than half of respondents indicated that building or repairing pipelines took the most time during the day.

When data incidents occurred, the majority of data professionals – 62% – said their time to detection was five to eight hours. The average time to resolve data incidents was nine hours.

Data quality and the bottom line

Poor data quality also appeared to affect the bottom line: 47% of respondents estimated that poor data affected 25% or more of their business revenue.

Of the 300 data professionals surveyed:

• 75% takes four hours or more to detect a data quality incident.
• About half said it takes an average of nine hours to fix the problem after it is identified.
• 58% said the total number of incidents has increased slightly or significantly in the past year, often due to more complex pipelines, larger data teams, larger amounts of data and other factors.

However, there is also positive news. About 90% of respondents said they were already investing or planning to invest in data quality solutions, such as data observation, within six months.

The survey of more than 300 data professionals was conducted between April 28 and May 11, 2022, using an email invitation and an online survey.

Read the full report from Monte Carlo.