Principal Databricks Data Engineer
Job Location:
Monthly Salary:
Experience Required:
Posted on:
4 hours ago
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Principal Databricks Data Engineer
Experience Required: 1218 Years
Key Requirements
1218 years of overall Data Engineering experience
8 years of experience with enterprise Data Warehouse and Data Lake platforms
5 years of hands-on experience with Databricks and Apache Spark at scale
Strong experience modernizing legacy Cloudera platforms including:
CDH/CDP
Hive
HBase
Impala
Spark
Modernize Cloudera platforms to Databricks Lakehouse architecture
Redesign ingestion transformation and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation audit integrity and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Implement layered data architecture including:
Raw / Landing Layer
Curated / Conformed Layer
Semantic / Consumption Layer
Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
Strong experience with finance and risk data models including:
General Ledger
Sub-ledger
Financial Hierarchies
Credit Risk
Liquidity Risk
Market Risk
Enable reporting capabilities including:
Aggregation
Drill-down
Drill-back
Build and manage semantic and consumption layers for BI reporting and analytics
Define business metrics dimensions hierarchies and KPIs
Experience with:
Databricks SQL
Delta Tables
dbt or similar frameworks
Develop and optimize large-scale data pipelines using:
PySpark
Spark SQL
Delta Lake
Implement Medallion Architecture including:
Bronze Layer
Silver Layer
Gold Layer
Optimize workloads using:
Z-ORDER
OPTIMIZE
Caching
Cluster Configuration Tuning
Implement:
Data Governance
Data Quality Frameworks
Reconciliation Controls
Exception Handling
Establish data lineage and metadata management
Ensure data security access control and compliance standards
Experience with AWS or Azure cloud platforms
Experience with CI/CD pipelines using:
Git
Terraform
Jenkins
Azure DevOps
Familiarity with:
Apache Airflow
Databricks Workflows
Experience with dbt is an advantage
Act as a technical authority and lead enterprise architecture decisions
Mentor senior engineers and establish engineering standards
Collaborate with finance risk analytics and governance stakeholders
Translate complex data structures into business-ready insights
Nice-to-Have Skills
Experience in BFSI Capital Markets or Regulatory Reporting
Exposure to:
SAP Finance
Oracle Financials
SAP S/4HANA
Experience supporting AI/ML workloads
Databricks and Cloud Certifications
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory management and analytical reporting systems
Essential Skills
Databricks
Apache Spark
PySpark
Spark SQL
Delta Lake
Lakehouse Architecture
Cloudera (CDH/CDP)
Hive
HBase
Impala
Data Warehouse & Data Lake
Medallion Architecture
Databricks SQL
dbt
AWS / Azure
Airflow / Databricks Workflows
CI/CD
Terraform
Jenkins
Git
Finance & Risk Data Models
Enterprise Data Architecture
Required Skills:
Sailpoint
