Senior Data Engineer

Job Summary

Owning the Databricks pipelines for the Sugar Predict platform, the full-time Senior Data Engineer will drive production reliability, cost efficiency, and platform growth while working remotely with cross-functional teams to ensure high-quality data delivery at scale.

Key responsibilities

• Manage Databricks production support, including monitoring, alerting, and incident response for all data flows
• Optimize data pipelines to reduce compute costs and improve throughput while maintaining SLA performance metrics
• Migrate legacy ETL/ELT pipelines to Databricks and support customer onboarding by provisioning reliable data pipelines

Required qualifications

• 4+ years of data engineering experience
• At least 2 years of experience with Databricks or the Apache Spark ecosystem across Azure and/or AWS
• Proficiency in PySpark, SQL, and Python for building production-grade pipelines
• Hands-on experience with Delta Lake, including schema design and incremental processing
• Solid knowledge of PostgreSQL, including query optimization and schema design