Job VC
Data Quality Automation Engineer
Technologies
Description
About the project:
The Client provides comprehensive operational support and a range of expert services to the world’s leading insurers, brokers, fleet managers, and automotive manufacturers. 3,300 employees across ten countries deliver exceptional standards on a large scale for over 1,200 clients. We help the global insurance market to handle millions of claims each year in the most cost-effective and efficient ways possible.
The Client is embarking on an exciting and challenging transformation program, and our software solutions are a driving force behind this strategy, using cloud computing and leading-edge design patterns.
Key Responsibilities
Define and implement data quality rules across ingestion, transformation, and reporting layers
Validate data in Databricks-based pipelines
Monitor and test Databricks transformations (PySpark/SQL) for correctness and completeness
Ensure Databricks / Power BI reports reflect accurate and reconciled data
Set up data validation checks (schema, nulls, duplicates, ranges, referential integrity)
Identify, log, and track data quality issues with root cause analysis
Collaborate with data engineers and analysts to fix issues
Build automated data quality monitoring and alerts
Required Skills
4-5+ years of Relevant work experience in data analysis, quality assurance, data governance, or a similar field is highly desirable.
Strong knowledge of Databricks / Spark (SQL, PySpark)
Understanding of ETL/ELT pipelines and data transformations (dbt)
Experience validating BI/reporting outputs (Power BI preferred)
SQL proficiency for data validation and reconciliation
Familiarity with data quality frameworks/tools (e.g., Great Expectations is a plus)
Nice to Have
Experience with AWS data stack
Experience with data governance or data catalog tools
Exposure to CI/CD for data pipelines
Knowledge of data lineage and observability tools
Success Criteria
Reduced data defects in pipelines and reports
Automated data quality checks are in place
Clear visibility and tracking of data issues
The Client provides comprehensive operational support and a range of expert services to the world’s leading insurers, brokers, fleet managers, and automotive manufacturers. 3,300 employees across ten countries deliver exceptional standards on a large scale for over 1,200 clients. We help the global insurance market to handle millions of claims each year in the most cost-effective and efficient ways possible.
The Client is embarking on an exciting and challenging transformation program, and our software solutions are a driving force behind this strategy, using cloud computing and leading-edge design patterns.
Key Responsibilities
Define and implement data quality rules across ingestion, transformation, and reporting layers
Validate data in Databricks-based pipelines
Monitor and test Databricks transformations (PySpark/SQL) for correctness and completeness
Ensure Databricks / Power BI reports reflect accurate and reconciled data
Set up data validation checks (schema, nulls, duplicates, ranges, referential integrity)
Identify, log, and track data quality issues with root cause analysis
Collaborate with data engineers and analysts to fix issues
Build automated data quality monitoring and alerts
Required Skills
4-5+ years of Relevant work experience in data analysis, quality assurance, data governance, or a similar field is highly desirable.
Strong knowledge of Databricks / Spark (SQL, PySpark)
Understanding of ETL/ELT pipelines and data transformations (dbt)
Experience validating BI/reporting outputs (Power BI preferred)
SQL proficiency for data validation and reconciliation
Familiarity with data quality frameworks/tools (e.g., Great Expectations is a plus)
Nice to Have
Experience with AWS data stack
Experience with data governance or data catalog tools
Exposure to CI/CD for data pipelines
Knowledge of data lineage and observability tools
Success Criteria
Reduced data defects in pipelines and reports
Automated data quality checks are in place
Clear visibility and tracking of data issues