About the Role:

Grade Level (for internal use):

We are seeking ahands-on Senior Lead Data Engineer / Data Platform Leadto support our offshore team in India. This role focuses on executing data pipeline onboarding migration and support activities within our enterprise data platform. You will work closely with the Databricks team and internal stakeholders to ensure smooth transition ongoing refinement and reliable operation of data pipelines and integrations in aDatabricks-on-AWS lakehouse environment. Databricks on AWS positions data engineering around governed ingestion transformation and delivery of high-quality data for analytics and downstream use cases.

This is a delivery-focused role requiring strong technical expertise in data engineering with responsibilities for supporting data onboarding pipeline maintenance platform support and cloud-native operational excellence. You will lead a small team of engineers providing technical guidance and ensuring high-quality execution across scalable secure and governed data workflows leveragingAmazon S3AWS GlueAWS IAM andAWS Lake Formationalongside Databricks capabilities such asDelta LakeUnity Catalog and open table format interoperability where appropriate.

Key Responsibilities

Data Pipeline Support & Migration

• Assist in the migration of existing data pipelines to the Databricks platform following established patterns and runbooks.
• Support the ongoing operation troubleshooting and optimization of pipelines post-migration.
• Implement and follow engineering best practices for data ingestion transformation and monitoring.
• Ensure pipelines are reliable performant and maintainable.
• Support data ingestion and storage patterns usingAmazon S3as the core cloud object store for lakehouse data layers with Databricks processing and transformation built on top of that storage foundation.
• Contribute to migration and optimization activities involvingAWS Gluefor metadata cataloging schema discovery or ETL interoperability where required across the enterprise AWS data ecosystem.
• Support analytical integration patterns with downstream platforms such asAmazon Athenawhen business consumers require governed access beyond Databricks-native consumption paths.

Data Onboarding & Asset-Agnostic Support

• Support offshore teams in onboarding new data assets to the enterprise platform using standardized asset-agnostic processes.
• Help teams prepare validate and publish data assets ensuring consistency and compliance with platform standards.
• Document onboarding procedures and assist in resolving onboarding issues.
• Help standardize onboarding intoBronze Silver and Golddata layers and support practical lakehouse patterns for ingestion curation and consumption in Databricks on AWS.
• Support metadata-driven onboarding workflows usingAWS Glue Data Catalogand Databricks governance capabilities to improve discoverability schema consistency and reusable ingestion patterns.
• Assist with secure data onboarding by aligning access permissions and environment controls throughAWS IAMandAWS Lake Formation alongsideUnity Catalogpermissions and governance policies for managed access to datasets.

Platform Integration Support

• Support integration of data pipelines with the enterprise data mastering platform.
• Assist in data quality checks metadata management and reconciliation activities.
• Help troubleshoot and resolve issues related to data ingestion and mastering workflows.
• Support interoperability across AWS and Databricks services for ingestion transformation and governed publishing of mastered or standardized datasets.
• Contribute to table design and data publication patterns usingDelta Lakeand where applicableApache Icebergto support interoperable lakehouse consumption models.
• Assist with auditability and lineage-oriented controls through metadata access management and governed publishing standards.

Team Collaboration & Delivery

• Work closely with the offshore team to prioritize tasks meet delivery deadlines and maintain quality.
• Provide technical guidance review code and mentor junior team members.
• Communicate progress risks and issues effectively to stakeholders.
• Collaborate with platform governance and cloud teams to align engineering delivery with enterprise AWS standards security controls and support processes.
• Promote practical engineering discipline across source control peer review CI/CD release management and environment consistency.

Operational Excellence

• Follow established operational procedures for deployment monitoring and incident management.
• Support production support activities including issue resolution and performance tuning.
• Contribute to documentation and knowledge sharing.
• Support cloud-native monitoring and operational visibility using tools such asAmazon CloudWatch logging alerting and job observability for production data pipelines.
• Contribute to deployment automation and release reliability usingCI/CD infrastructure as code and cloud-native orchestration patterns including AWS services such asAWS Step FunctionsAWS Lambda or equivalent enterprise tooling where applicable.
• Help maintain secure and compliant platform operations through access control secrets handling environment segregation and operational readiness practices.

Required Qualifications

• 8 yearsof experience in data engineering pipeline development or related roles.
• Proven experience supporting data onboarding migration and pipeline operations.
• Hands-on expertise with data engineering tools such asSpark SQL Python or similar.
• Experience working withDatabricksor similar cloud data platforms.
• Hands-on experience with relevantAWS data engineering services especiallyAmazon S3AWS Glue and core cloud security concepts such asAWS IAM; familiarity withAWS Lake FormationAmazon AthenaAWS Lambda orAmazon Kinesisis strongly preferred for enterprise-scale data platform support. Industry-standard AWS data engineering stacks commonly center on these services for storage ETL governance analytics and streaming.
• Familiarity with data quality metadata lineage and basic data governance concepts.
• Working knowledge ofDelta LakeDatabricks Unity Catalog and exposure toApache Icebergor similar open table/storage formats for lakehouse architectures.
• Understanding of secure data access governance and platform controls across Databricks and AWS environments.
• Familiarity with software engineering best practices including version control automated testingCI/CD infrastructure as code and monitoring.
• Strong problem-solving skills and ability to troubleshoot production issues.
• Good communication skills and ability to collaborate with global teams.

Preferred Qualifications

• Experience supporting enterprise data onboarding and integration workflows.
• Knowledge ofDelta Lake Apache Iceberg or similar storage formats.
• Exposure to data mastering or data governance frameworks.
• Experience withAWS Lake FormationAWS Glue Data CatalogAmazon AthenaAmazon Kinesis orAWS Step Functionsin support of modern cloud data platforms.
• Experience supportingDatabricks on AWSlakehouse implementations including governed data access withUnity Catalogand cloud storage patterns built onAmazon S3.
• Proficiency inPythonandSQL with experience inPySparkor similar distributed processing frameworks.
• Familiarity with infrastructure as code and cloud deployment automation.
• Certifications inDatabricksAWS or related data platforms.