Senior Databricks DevOps Engineer (Azure & CI/CD)

Experience Required

7 Years

Job Summary

We are seeking an experienced Databricks DevOps Engineer with strong expertise in Azure Cloud Azure Databricks CI/CD automation Infrastructure as Code (IaC) and cloud-native application deployment. The ideal candidate will have hands-on experience in Azure platform services Databricks administration Terraform module development API integrations and DevOps automation.

The candidate will play a key role in designing automating deploying and supporting scalable cloud and data platform solutions while enabling continuous delivery and operational excellence across enterprise environments.

Key Responsibilities

• Design develop and maintain Azure Databricks environments and cloud platform infrastructure.
• Build and manage CI/CD pipelines using Jenkins GitLab CI/CD and DevOps best practices.
• Develop reusable Infrastructure as Code (IaC) solutions using Terraform and Terraform modules.
• Automate deployment monitoring and operational processes across Azure and Databricks environments.
• Design and implement secure authentication and authorization solutions using Azure AD OAuth JWT and Azure AD B2C.
• Develop and integrate REST APIs and third-party APIs with enterprise applications.
• Manage Azure application registrations service principals and identity integrations.
• Deploy and maintain cloud-native applications on Azure Kubernetes Service (AKS).
• Build and support Microsoft Teams applications using Microsoft Teams SDK and Bot Framework.
• Implement real-time communication solutions using Azure SignalR Services.
• Configure monitoring logging and observability using Azure Application Insights and Azure monitoring tools.
• Collaborate with Data Engineering teams to support ETL/ELT workloads and data platform operations.
• Ensure security scalability reliability and performance of cloud-based solutions.
• Participate in troubleshooting incident resolution and root cause analysis activities.
• Support Agile development and DevOps delivery processes.

Required Skills

Cloud & Platform Technologies

• Microsoft Azure
• Azure Databricks
• Azure Active Directory (Azure AD)
• Azure AD B2C
• Azure Kubernetes Service (AKS)
• Azure SignalR
• Azure Application Insights

DevOps & Automation

• CI/CD Pipelines
• Jenkins
• GitLab CI/CD
• Infrastructure as Code (IaC)
• Terraform
• Terraform Module Development
• Git
• GitHub
• GitLab

Security & Identity

• OAuth 2.0
• JWT Authentication
• Azure App Registrations
• Service Principals
• Identity & Access Management

Development & Integration

• REST APIs
• Third-Party API Integrations
• SQL Server
• Oracle SQL
• Python
• Bash Scripting
• PowerShell

Collaboration Platforms

• Microsoft Teams SDK
• Teams Application Development
• Azure Bot Framework

Nice to Have Skills

Data Engineering

• Data Engineering Concepts
• ETL / ELT Development
• Data Warehousing
• Data Modeling
• Big Data Technologies
• Data Pipeline Automation

Additional Skills

• Workflow Automation
• Monitoring & Observability
• Cloud Governance
• Security Best Practices

Preferred Qualifications

• Experience supporting enterprise-scale Azure Databricks environments.
• Hands-on experience with Terraform module development and cloud automation.
• Experience integrating Azure services with enterprise applications.
• Exposure to Data Engineering and modern data platforms.
• Knowledge of DevSecOps practices.

Soft Skills

• Strong analytical and troubleshooting abilities.
• Excellent communication and stakeholder management skills.
• Ability to work independently in fast-paced environments.
• Strong collaboration and problem-solving mindset.