Senior Databricks Data Engineer

Job Summary

Focused on transforming spreadsheet and flat-file data into governed Delta tables, the full-time Senior Databricks Data Engineer will build ingestion pipelines, ensure data quality, and collaborate with business owners to document and validate data structures in a fully remote environment.

Key responsibilities

• Inventory and document Excel workbooks and flat-file sources used by business and engineering teams
• Build Auto Loader pipelines for ingesting data into S3, ensuring schema adherence and data quality
• Translate spreadsheet logic into maintainable SQL and document business rules for consistency

Required qualifications

• Production experience building ingestion pipelines on Databricks
• Hands-on experience with Excel and flat-file ingestion into Spark or Databricks
• Strong knowledge of PySpark and SQL, with experience in Delta Lake and medallion architecture
• Experience working directly with non-technical data owners to define business rules
• U.S. citizenship and fluent English, both written and spoken