Senior Databricks Data Engineer

Job Summary

To ensure data integrity for a large-scale data platform, the full-time Senior Databricks Data Engineer will build automated testing frameworks using PySpark and SQL, focusing on reconciliation tests, incremental loading, and data quality validation in a fully remote environment.

Key responsibilities

• Build and run automated test frameworks for data pipelines across all environments
• Write reconciliation tests to compare Databricks tables with legacy systems and reports
• Define data quality expectations in pipelines to ensure bad data is managed effectively

Required qualifications

• 5+ years of experience in data engineering with a focus on test and validation frameworks
• Strong proficiency in PySpark and SQL for writing reconciliation and assertion logic
• Hands-on experience reconciling new platforms against legacy systems
• Deep understanding of incremental load correctness and data quality management
• Experience with Python test frameworks and data testing tools