Unit
- Data quality definition: Data quality is the degree to which data meets the requirements of its intended use. It is not an intrinsic property but is context-dependent. For example, customer addresses accurate for shipping may be sufficient even if they lack postal codes for marketing analytics.
- Business consequences of poor quality: Low-quality data leads to incorrect decisions, wasted resources, and compliance penalties. A single duplicate customer record can cascade into redundant mailings, misallocated budgets, and skewed customer lifetime value calculations.
- Cost of poor data quality: Research indicates that organizations typically waste 15-30% of their revenue on data quality issues. This includes costs from rework, manual corrections, lost sales, and regulatory fines. For example, a financial institution may incur millions in penalties for misreported transactions.
- Data quality in different sectors: In healthcare, incorrect patient records can endanger lives. In e-commerce, wrong inventory data causes stockouts. In government, census errors affect resource allocation. Each sector has unique quality requirements and tolerance thresholds.
- The data lifecycle and quality: Quality must be managed at all stages: collection, storage, processing, analysis, and archival. Poor quality at any stage propagates downstream, amplifying errors. A sensor reading error at capture may become undetectable after aggregation.
- Common data quality issues: Frequent problems include missing values, inconsistencies (e.g., date formats), duplicates, outdated records, and violations of business rules. For instance, a customer age of 200 years violates domain constraints.
- Data quality vs. data integrity: Data integrity focuses on accuracy and consistency over the data's lifecycle, often via constraints and validations. Data quality is broader, incorporating fitness for use, timeliness, and relevance. Both are essential but address different aspects.
- ROI of data quality management: Investing in data quality yields returns through improved decision-making, operational efficiency, and customer satisfaction. A case study shows a retail chain reduced return rates by 10% after cleaning product master data.
- Data quality dimensions: Data quality is measured across multiple dimensions: accuracy (data correctly represents reality), completeness (all required data present), consistency (no contradictions across sources), timeliness (data is current), validity (data conforms to defined rules), and uniqueness (no duplicate records). Each dimension requires specific monitoring and improvement techniques.
- Data quality measurement: Organizations track data quality through metrics like completeness percentage, accuracy rate, and timeliness lag. These metrics are often displayed on dashboards for ongoing visibility. Establishing baselines and setting improvement targets enables teams to quantify progress. Regular audits and profiling help identify emerging issues before they impact business processes.
- Data governance and quality: Data governance provides the framework for data quality management by defining roles, policies, and standards. A data steward oversees quality initiatives, ensures accountability, and enforces rules. Without governance, quality efforts become fragmented and unsustainable. Governance also facilitates cross functional collaboration to resolve systemic data issues.
Code Example
SQL
-- Detect missing values in a customer table
SELECT COUNT(*) AS total_rows,
COUNT(email) AS non_null_emails,
COUNT(phone) AS non_null_phones
FROM customers;This query profiles completeness by counting rows with non-null columns. A low non-null count indicates quality issues requiring remediation.
Quick Revision
- Data quality is defined as fitness for use, varying by context.
- Poor quality costs businesses 15-30% of revenue in wasted resources.
- Quality issues propagate through the data lifecycle, so early detection is critical.
- Common problems include missing values, duplicates, and inconsistencies.
- Investing in quality management yields measurable ROI through better outcomes.