Lead Manager Data Engineering

We are looking for an experienced Manager Data Engineering with 8 years of experience in building and managing data engineering solutions in an on-premise environment. The ideal candidate should have strong hands-on expertise in Python Apache Spark SQL and Apache Airflow along with proven experience in leading data engineering teams and delivering scalable data pipelines.

Key Responsibilities

• Lead and manage a team of Data Engineers and provide technical guidance and mentorship.
• Design develop and optimize scalable data pipelines in an on-premise environment.
• Build robust ETL/ELT workflows using Python Spark SQL and Airflow.
• Design and manage complex Apache Airflow DAGs for data pipeline orchestration.
• Develop and optimize Spark-based data processing solutions for large datasets.
• Write complex and optimized SQL queries for data extraction transformation and analysis.
• Troubleshoot pipeline failures performance issues and data quality challenges.
• Work closely with architects business stakeholders and cross-functional teams to understand requirements and deliver solutions.
• Conduct code reviews and ensure adherence to engineering and development best practices.
• Drive technical design estimation planning and end-to-end project delivery.
• Monitor team performance project timelines risks and dependencies.
• Mentor engineers and contribute to building a strong data engineering practice.

Qualifications :

• 8 years of overall experience in Data Engineering.
• Strong hands-on experience with Python.
• Strong experience with Apache Spark / PySpark.
• Advanced SQL skills including complex joins CTEs window functions subqueries and query optimization.
• Strong experience with Apache Airflow for workflow orchestration and scheduling.
• Experience working with on-premise data environments.
• Strong understanding of ETL/ELT processes and data pipeline architecture.
• Experience handling large volumes of data and optimizing data processing workloads.
• Good understanding of data quality monitoring troubleshooting and performance optimization.

Leadership Requirements

• Proven experience managing or leading Data Engineering teams.
• Strong stakeholder and client management skills.
• Ability to provide technical direction while managing project delivery.
• Experience with resource planning task allocation mentoring and performance management.
• Strong communication and problem-solving skills.
• Ability to work in a fast-paced Agile environment.

Good to Have

• Experience in large-scale enterprise data platforms.
• Experience with data warehouse concepts and data modelling.
• Experience with on-premise Hadoop/data ecosystems.
• Experience in migrating or modernizing legacy/on-premise data platforms.

Remote Work :

No

Employment Type :

Full-time