Senior Databricks Data Engineer

Job Summary

To support a growing data platform initiative, the full-time Senior Databricks Data Engineer will manage the ingestion of Excel and flat-file sources into governed Delta tables, ensuring data quality and lineage while collaborating with business owners remotely.

Key responsibilities

• Inventory and document the structure of Excel workbooks and flat-file sources used by business and engineering teams
• Build Auto Loader pipelines for ingesting data from S3, including schema evolution and data quality checks
• Translate spreadsheet logic into maintainable SQL and validate that platform outputs align with trusted user data

Required qualifications

• Production experience building ingestion pipelines on Databricks
• Hands-on experience with ingesting Excel and flat-file sources into Spark or Databricks at scale
• Strong proficiency in PySpark and SQL, with knowledge of Delta Lake and medallion architecture
• Experience with Auto Loader for incremental file ingestion from S3
• US citizenship and fluent English, both written and spoken