Lead Software Engineer Databricks

Be an integral part of an agile team thats constantly pushing the envelope to enhance build and deliver top-notch technology products.

As a Lead Software Engineer-Databricks at JPMorgan Chase within our Corporate Sectors Enterprise Technology team you are an integral part of an agile team that works to enhance build and deliver trusted market-leading technology products in a secure stable and scalable way. As a core technical contributor you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

Job responsibilities

• Lead the architecture and delivery of high-throughput low-latency data pipelines on Databricks using Apache Spark (Core SQL Structured Streaming) driving performance reliability and scalability.
• Establish and evolve Lakehouse patterns with Delta Lake (ACID transactions schema evolution time travel Z-ordering compaction) to ensure performant maintainable data platforms at scale.
• Own Databricks cluster strategy and configuration including runtime selection autoscaling driver/executor sizing Spark configurations init scripts cluster policies pools and instance profiles.
• Orchestrate and automate pipelines and jobs using Databricks Workflows integrating with AWS eventing and orchestration services as needed.
• Design secure ingestion and transformation frameworks leveraging Databricks services including Delta or unmanaged table design ingestion task creation and Airflow DAGs to produce trusted and refined datasets.
• Enforce data quality lineage and governance using Unity Catalog and/or AWS Glue Catalog embedding expectations and validation directly into pipelines.
• Drive Spark and Databricks performance engineering and tuning (partitioning and file sizing AQE broadcast joins shuffle tuning caching spill/memory control job right-sizing and liquid clustering/partitioning keys) to optimize cost and throughput.
• Build and maintain reusable libraries frameworks and APIs in Python and/or Java ensuring strong unit integration and data validation test coverage.
• Implement CI/CD for data projects using Git-based workflows Terraform-based infrastructure deployments and environment promotion and automated releases; champion engineering standards code reviews and enterprise-authorized AI-assisted engineering practices (e.g. code review/refactoring test acceleration and incident/root-cause analysis) with consistent validation (secure coding peer review automated testing) and reuse of proven patterns.
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality delivery speed and operational outcomes (e.g. AI-assisted code review/refactoring test strategy acceleration incident/root-cause analysis support) while establishing consistent validation standards (secure coding peer review automated testing) and promoting reuse of effective patterns across the team.

• Applies knowledge of tools within the Software Development Life Cycle toolchain including enterprise-authorized AI-assisted development and automation capabilities to improve the value realized by automation.
  

  Required qualifications capabilities and skills

• Formal training or certification on software engineering concepts and 5 years applied experience.
• Advanced experience in software engineering and data engineering including significant production delivery with Apache Spark on Databricks and/or AWS EMR.
• Advanced hands-on Databricks expertise across Delta Lake Unity Catalog Workflows Repos/notebooks and SQL Warehouses including cluster configuration and optimization.
• Proven ability to architect build and operate reliable ETL/ELT data pipelines (batch and streaming) including schema design/evolution SLAs and reliability engineering practices.
• Deep Spark performance tuning skills with experience diagnosing bottlenecks and optimizing jobs for scalability cost and runtime efficiency.
• Strong programming proficiency in Python and/or Java for data processing platform tooling and automation.
• Strong SQL and analytics data modeling expertise including dimensional/star schema design and Lakehouse best practices.
• Demonstrated experience leading effective use of approved AI-assisted software development tools (coding code review test acceleration troubleshooting) including setting team expectations and validation standards for correctness performance and security of AI outputs.
• Strong responsible-AI and security-first engineering mindset including data sensitivity awareness secure handling of inputs/outputs roles/instance profiles secrets management encryption at rest/in transit network controls and adherence to resiliency and security expectations; experience coaching teams on safe compliant adoption within delivery practices.
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g. for coding code review test acceleration troubleshooting) with the ability to set team expectations for validating AI outputs for correctness performance and security.
• Strong understanding of responsible AI use in engineering workflows including data sensitivity considerations secure handling of inputs/outputs and adherence to resiliency and security expectations; experience coaching engineers on safe compliant adoption within delivery practices
  
  Preferred qualifications capabilities and skills
• Experience with Delta Live Tables and advanced governance (catalogs grants auditing) in Databricks.
• AWS networking knowledge (VPC subnets routing security groups) and data egress controls.
• Experience with Terraform for Infra deployments
• Cost optimization experience: autoscaling strategies spot vs on-demand auto-termination storage layouts and compaction.
• Familiarity with Airflow Genie Streamlit and React
• Observability for data systems (freshness/completeness metrics lineage SLAs alerting).
• Demonstrated leadership in code quality reviews testing strategy CI/CD and technical mentorship; excellent communication with stakeholders.