Big Data Lead

Job Summary

To support data-driven decision-making, the full-time Big Data Lead will design, implement, and optimize ETL/ELT pipelines while collaborating with stakeholders and ensuring data quality in a remote environment.

Key responsibilities

• Design, implement, and optimize end-to-end ETL/ELT pipelines for large volumes of data
• Write scalable code in Python and Pyspark, and review existing code for improvements
• Collaborate with IT teams and business stakeholders to gather data requirements and troubleshoot data issues

Required qualifications

• Strong proficiency in Python, Pyspark, and SQL (Microsoft SQL Server preferred)
• Experience with big data frameworks such as Hadoop and Spark
• Familiarity with cloud platforms like AWS, Azure, or GCP
• Experience with code versioning tools such as Bitbucket or Github
• Solid understanding of database design principles and data warehousing solutions