7 data quality best practices to improve data performance | 7wData

• Read original article here

Data quality is essential for any analysis or business intelligence. Employing best practices lets organizations address issues that become even more critical and challenging as teams build a data analytics pipeline.

Subtle problems can get magnified by improved automation and increased data aggregation. Teams may also struggle to sort out the precise cause of issues buried within complex data pipelines.

"Data is often viewed as an organization's most valuable asset," said Alexander Wurm, analyst at Nucleus Research. "However, this is not always true. Poor data quality can taint business outcomes with inaccurate information and negatively impact operations rather than improving them."

Enterprises can set up practices to track data lineage, ensure data quality and protect against counterproductive data.

There are many aspects of data quality teams need to address. Teams should start with core attributes of high and low data quality, said Terri Sage, CTO of 1010data, a provider of analytical intelligence to the financial, retail, and consumer markets. These must reflect characteristics such as validity, accuracy, completeness, relevance, uniformity and consistency. Teams that automate these measurements can determine when their efforts are paying off. Additionally, these metrics can also help teams correlate the cost of interventions, tools or processes with their effect on data quality.

Why data quality is important for the data analytics pipeline Data quality is essential to the data analytics and data science pipeline. Low-quality data may lead to bad decisions, such as spending money on the wrong things, Sage said. Incorrect or invalid data can impact operations, such as falsely detecting a cyber security incident. : Seven best practices to improve data quality High data quality is measured by how well it has been deduplicated, corrected and validated and whether it has the correct key observations. High-quality data leads to better decisions and outcomes based on the fit for their intended purpose. In contrast, bad data can reduce customer trust and lower consumer confidence. Correcting data riddled with errors also consumes valuable time and resources. "An enterprise with poor-quality data may be making ill-judged business decisions that could lead to lost sales opportunities or lost customers," said Radhakrishnan Rajagopalan, global head for technology services at Mindtree, an IT consultancy.

There are various ways the data analytics pipeline impacts data quality. One of the biggest issues Sujoy Paul -- vice president of data engineering and data science at Avalara, a tax automation platform -- faces is the quality of data they are aggregating. Two factors make data quality challenging as they grow their data aggregation pipeline. One issue is potentially losing or duplicating data during transfer from source systems to data lakes and data warehouses. For example, memory issues with cloud data pipeline technologies and data queuing mechanisms often cause result in small batches of lost transactions. The second issue is unpredictable variations in the source systems leading to significant data quality issues in destination systems. Many potential problems lead to unpredictable data from source systems, but changes in data models, including small changes to data types, can cause significant variations in destination systems. Here are seven data quality best practices to improve performance: Teams should curate an accurate, digestible picture of data assets and pipelines, their quality scores, and detailed data lineage analysis, said Danny Sandwell, director of product marketing at Quest Software, an IT management software provider. This map identifies where data comes from and how it may change in transit. Many teams use data transformation to streamline integration. However, many advanced analytics require raw data to provide sufficient accuracy and detail.