Azure Databricks Platform Architect Technical Lead

Hybrid preferred however can be Pan India Remote for the right candidate

Azure Databricks Platform Architect / Technical Lead

About the Role

We are looking for an experienced Azure Databricks Platform Architect / Technical Lead with strong hands-on experience in enterprise data platform architecture infrastructure and Databricks development.

The ideal candidate should be able to design provision and manage enterprise-scale Databricks environments including networking security governance and platform operations.

Responsibilities:

Design and implement enterprise-scale Azure Databricks platform architecture including multi-workspace environments and Azure landing zones.

Provision and configure Databricks workspaces clusters cluster policies and compute resources.

Design Azure networking solutions including VNets subnets NSGs CIDR planning IP allocation and secure connectivity.

Configure and manage Unity Catalog including metastore setup RBAC permissions data lineage and governance.

Implement platform security including Azure Key Vault secrets management credential rotation and auditing.

Automate infrastructure provisioning and deployments using Terraform Bicep ARM Templates and CI/CD pipelines.

Define platform standards for monitoring performance cost optimization and operational support.

Support Databricks development including PySpark Spark SQL Delta Lake and Medallion architecture.

Lead technical design decisions and collaborate with Data Engineering DevOps Infrastructure and Security teams.

Requirements
8 years of experience in Data Engineering Cloud Infrastructure or Platform Architecture including at least 3 years with Azure Databricks.
Hands-on experience designing and provisioning Azure Databricks workspaces from scratch.
Strong experience with enterprise Databricks architecture including multi-workspace environments and secure connectivity.
Strong Azure networking knowledge including VNets subnets NSGs CIDR planning routing and IP address management.
Hands-on experience with Unity Catalog metastore configuration RBAC access controls and data lineage implementation.
Experience with Azure security Key Vault secret rotation and credential lifecycle management.
Experience with Infrastructure as Code preferably Terraform and CI/CD automation.
Strong understanding of Databricks clusters cluster policies compute management and performance optimization.
Experience with enterprise data governance monitoring auditing and production support.
Good understanding of Python PySpark SQL Delta Lake and Medallion architecture.
Experience leading technical design and architecture discussions.

Nice to Have
Experience with multi-region Azure deployments and complex enterprise networking.
Experience with Databricks Asset Bundles (DABs).
Experience with Azure Monitor Log Analytics and cost governance.
Experience in banking insurance or other regulated industries.
Azure or Databricks certifications.

Required Experience:

Staff IC