Data Integration Lead

Job Summary

To support data-driven initiatives, the full-time Data Integration Lead will design and implement scalable data pipelines using Databricks, focusing on ETL processes and optimizing data workflows while ensuring compliance with data privacy regulations.

Key responsibilities

• Design and implement scalable data pipelines for ingesting and transforming data using Databricks and PySpark
• Analyze existing integration landscape and define ingestion and integration strategies for Databricks
• Collaborate with stakeholders to gather requirements and create technical solutions while documenting workflows and architecture

Required qualifications

• Experience with Databricks, PySpark, and Spark SQL
• Proficiency in ETL processes and data lakehouse architecture
• Familiarity with data integration tools such as Teradata, Talend, and IBM Sterling
• Understanding of data security and compliance regulations
• Ability to create comprehensive documentation of data workflows and architectures