Data Quality Management

Accurate analyses and sound decisions thanks to clean data

What Is Data Quality and Data Quality Management?

Data quality is a subjective term that must be defined individually for each company. It refers to the overall characteristics of a dataset that enable it to meet user requirements.
Data quality management refers to all processes and procedures involved in ensuring high data quality. This includes identifying, cleansing, and making data available.

The Main Reasons for Poor Data Quality

Data is never 100 percent clean and perfect. This may be because data enters a company through various channels. As a result, it may be outdated, duplicated, or inconsistent. To ensure the highest possible data quality, it helps to understand the main causes of data errors. This way, you can prevent poor data quality before it happens. Here are the main causes:

• Manual Data Entry: Many companies rely on manual data entry. However, this process is highly prone to errors. Data can be entered in the wrong place or in the wrong format, and typos or transposed digits can easily occur.
• Data conversion: When transferring data from one storage location to another, data can be accidentally lost or altered. This can happen, for example, because the data is stored in different formats or the data structure is different.
• Real-time updates: To make sound decisions, it is important to work with up-to-date data at all times. However, errors can still occur if individual data records have not yet been updated at the time of analysis or if there has not been enough time to verify the data.
• Data Merging: When data needs to be merged—for example, during consolidations, mergers, or system changes—errors such as invalid formats, duplicates, and conflicts can also occur.
• System Upgrades: Frequent updates or upgrades to your software can also lead to errors, as data may be deleted or corrupted during the process.
• Indiscriminate Data Collection: Companies often collect all the data they generate. This offers some potential, as the data might be needed in the future. However, it also makes quality assurance and data analysis more difficult. Therefore, only the data that is truly needed should be stored whenever possible.

How can data quality be determined? The criteria

Various criteria show you how high the quality of your data is and whether the data is suitable for a particular task.

• Completeness: Are all the required data records complete?
  Incomplete data may be unusable or only partially usable. Therefore, it is essential to ensure that a data record contains all necessary attributes and that those attributes, in turn, contain all necessary data.
• Relevance: Is all the data required for the planned purposes available?
  Not all data collected is relevant to your purposes. Therefore, it should be collected deliberately, so that only as much data as necessary is gathered. This applies particularly to customer data, which is subject to data protection regulations.
• Accuracy: Is the collected data correct and reported as required?
  When collecting data, it is important to ensure that the data is accurate. At the same time, it should also be available at the required level of detail. This means, for example, that all necessary decimal places should be recorded.
• Timeliness: Are the data sets up to date?
  New data is constantly being generated within a company. Therefore, it makes sense to always perform analyses using current data in order to identify changes or problems early on. In practice, we often recommend that our clients base their decision-making on data from a reliable point in time. Depending on the situation, it may make sense, for example, to use data from the previous day, since live data can change very quickly.
• Validity: Is the origin of the data reliable, or do the data come from reliable sources?
  The origin of the data sets should be traceable in order to assess whether the data are reliable.
• Availability and Accessibility: Can users easily access the data they need? Is it available in the required format?
  For example, if relevant data is scattered across various tools or is not available in the correct format, easy accessibility is not always guaranteed.
• Consistency: Are there any contradictions or duplicates in the data? Are there any discrepancies with other data?
  Data must be unique, free of contradictions with itself or other data, free of redundancies, and structured consistently.

What can be done to ensure high data quality?

To ensure that data consistently maintains a high level of quality, the first step is to define how its quality can be measured. The data should then be analyzed, cleaned, and monitored based on the defined criteria. This process should be carried out regularly to maintain consistently high data quality and to permanently eliminate sources of error.

1. Define criteria

The first step is to determine which criteria should be used to measure data quality. For example, you’ll define what data must be available for your purposes and in what format it should be provided.

2. Data profiling / data analysis:

Data analysis is used to identify duplicate data, inconsistencies, errors, and incomplete information. This allows the quality of the data to be measured and, in subsequent steps, the data to be cleaned and updated. In addition, data analysis can be used to identify sources of error and thus take measures to ensure that the identified errors do not occur again in the future.

3. Data Cleaning

The Data Cleaning step resolves the issues identified during data analysis. This means that duplicates are deleted, incomplete data is filled in, and inconsistencies are corrected.

4. Data Monitoring:

The existing and new data should be continuously checked to ensure high data quality on a permanent basis.

Tips for data quality management:

1. Determine responsible persons

Without someone taking responsibility for data quality, no one may feel accountable for it. That is why it is important to designate responsible parties. Depending on the dataset, these may be different people, or they may be a single employee. Those responsible are tasked with ensuring that the defined standards are followed when creating the data and that the data is regularly reviewed and maintained.

2. Dealing with quality deficiencies

There is no such thing as 100 percent data quality, since errors can occur at any time. Depending on the intended use, however, it is possible to determine which data must be accurate in order to perform correct analyses and thus make the right decisions.
Our tip: While it is important that as many data records as possible are accurate, However, the cost-benefit ratio of making corrections may be unfavorable—for example, if cleaning the data takes a lot of time but you end up using the data very little afterward or it has no relevance. Therefore, prioritize addressing quality issues in the essential data.

3. Improve data quality directly at the source

Business intelligence solutions such as myPARM BIact allow you to manually modify, correct, or supplement stored data. However, you should keep in mind that, on the one hand, the data source remains incorrect even after such corrections, and on the other hand, manual corrections also carry a high risk of error. Furthermore, existing errors might be overlooked. Therefore, the quality of the data should ideally be improved at the data source itself. This ensures that high-quality data is made available to the BI software.

4. Continuous data monitoring

The more often you identify errors, correct them, and take action to address them, the higher the quality of your data will be in the future. Nevertheless, it is important to view data quality as an iterative process, since new errors can arise at any time, data requirements can change, or the volume and diversity of data can increase. The data quality management process should therefore be carried out on an ongoing basis.

Conclusion

Making decisions based on data rather than gut feelings can greatly contribute to your company’s success. However, this comes with the risk that the data leading to a decision may be inaccurate. For this reason, it is important to have a robust data quality management system in place to ensure that you can always rely on the accuracy of your data.

Would you like to get to know myPARM BI^act in a demo presentation? Then make an appointment right away!