Data Engineer

We are looking for an experienced Data Engineer to design develop and maintain scalable data pipelines and data processing solutions using Python PySpark Databricks and Microsoft Azure.

The ideal candidate should have strong hands-on experience in data engineering ETL/ELT development distributed data processing and cloud-based data platforms.

• Design develop and maintain scalable data pipelines and ETL/ELT workflows.
• Develop data transformation and processing solutions using PySpark and Python.
• Build and optimize data pipelines using Databricks.
• Work with Azure cloud services to develop and deploy data engineering solutions.
• Perform data ingestion transformation cleansing and integration from multiple sources.
• Optimize Spark jobs queries and data pipelines for performance and scalability.
• Implement data quality checks validation monitoring and error handling.
• Work with structured and semi-structured data.
• Collaborate with Data Analysts Data Scientists Architects and business teams to understand data requirements.
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Follow best practices for security governance version control and deployment.

• Strong hands-on experience in PySpark.
• Good programming experience in Python.
• Hands-on experience with Databricks.
• Experience working with Microsoft Azure cloud platform.
• Strong understanding of ETL/ELT concepts and data pipeline development.
• Good understanding of SQL and data processing concepts.
• Experience with distributed data processing using Apache Spark.

Required Skills:

Pythonfullstack