Databricks Engineer

We are looking for an experienced Azure Databricks Engineer with strong hands-on expertise in Python SQL and Apache Spark to design build and optimize scalable data pipelines and analytics solutions on the Azure cloud platform. The ideal candidate should have experience working with large datasets distributed data processing and modern data engineering practices.  

Responsibilities 

• Design develop and maintain scalable data pipelines using Azure Databricks 

• Implement ETL/ELT workflows using PySpark Spark SQL and Python 

• Optimize Spark jobs for performance cost and scalability 

• Work with structured and semi-structured data (Parquet Delta JSON CSV) 

• Build and manage Delta Lake tables (ACID time travel schema evolution) 

• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2) 

• Develop complex queries and transformations using SQL 

• Collaborate with data scientists analysts and stakeholders to support analytics and ML use cases 

• Ensure data quality validation and monitoring 

• Follow best practices for security access control and governance in Azure 

Qualifications :

• 4 years of experience in Data Engineering 

• Strong hands-on experience with Azure Databricks 

• Proficiency in Python for data processing 

• Strong knowledge of SQL (joins window functions performance tuning) 

• Hands-on experience with Apache Spark / PySpark 

• Experience working with Delta Lake 

• Knowledge of Azure Data Lake Storage (ADLS Gen2) 

• Understanding of distributed computing concepts 

• Experience with Git version control 

Additional Information :

• Experience with Azure Data Factory 

• Exposure to CI/CD pipelines (Azure DevOps GitHub Actions) 

• Basic understanding of data modeling 

• Familiarity with cloud security and RBAC in Azure 

• Exposure to streaming data (Spark Structured Streaming Event Hub Kafka) 

Remote Work :

Yes

Employment Type :

Full-time