Sr. Data Engineer (SQL Databricks PySpark)

Job: Senior Data Engineer

Location: Chicago IL (Hybrid 3 days Work from Office)

Experience: 10 Years

What is in it for you

Seeking a Senior Data Engineer with deep expertise in SQL Server Azure SQL and Databricks to design optimize and deliver scalable cloud-based data solutions supporting analytics and enterprise data platforms.

Roles and Responsibilities

• Design build and support end-to-end data pipelines (ingestion transformation validation publishing). Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
• Build production-grade Python components (reusable modules logging error handling testing). Create and maintain Azure Data Factory (ADF) pipelines (triggers parameterization monitoring failure handling).
• Work within Azure environments (ADLS Gen2 Azure SQL resource groups portal operations).
• Provision and maintain Azure components using Pulumi (Infrastructure as Code). Participate in code reviews documentation and operational support (triage root cause analysis)

Location: -

Chicago IL (Hybrid 3 days Work from Office)

Educational Qualifications: -

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
• Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

• ETL / ELT Concepts: Strong understanding of pipeline patterns incremental loads data validation and troubleshooting.
• SQL: Advanced querying (CTEs views joins complex query logic) and performance tuning for transformations and validation. Python: Production-quality development (modular code testing logging integration with APIs/files CICD Unit Test/Integration test automation Code Coverage).
• PySpark: Distributed transformations and performance optimization (joins partitions debugging) CICD Unit Test/Integration test automation Code Coverage.
• Azure Data Factory (ADF): Build/operate ADF pipelines parameterization triggers monitoring retry/error handling; integrate with Databricks/ADLS.
• Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration.
• Azure Fundamentals Pulumi: Hands-on with ADLS Gen2 Azure Portal Storage Explorer Resource Groups Azure SQL and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments
• Local to Chicago IL required as this needs candidates to go for in person client interview and 3 days Work from Office per Week.

Nice to have skills: -

• Ability to support/translate validation rules with SQL scripts and create data quality reports. TypeScript: Useful for pulumi pipeline to create Azure components.
• Java: Useful for integration with existing services/components.
• .NET: Useful for integration with existing services/components. Angular / Spring Boot: Minor troubleshooting or coordination with app teams.

Domain: Clinical (Health care)

Job: Senior Data Engineer

Location: Chicago IL (Hybrid 3 days Work from Office)

Experience: 10 Years

What is in it for you

Seeking a Senior Data Engineer with deep expertise in SQL Server Azure SQL and Databricks to design optimize and deliver scalable cloud-based data solutions supporting analytics and enterprise data platforms.

Roles and Responsibilities

• Design build and support end-to-end data pipelines (ingestion transformation validation publishing). Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
• Build production-grade Python components (reusable modules logging error handling testing). Create and maintain Azure Data Factory (ADF) pipelines (triggers parameterization monitoring failure handling).
• Work within Azure environments (ADLS Gen2 Azure SQL resource groups portal operations).
• Provision and maintain Azure components using Pulumi (Infrastructure as Code). Participate in code reviews documentation and operational support (triage root cause analysis)

Location: -

Chicago IL (Hybrid 3 days Work from Office)

Educational Qualifications: -

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
• Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

• ETL / ELT Concepts: Strong understanding of pipeline patterns incremental loads data validation and troubleshooting.
• SQL: Advanced querying (CTEs views joins complex query logic) and performance tuning for transformations and validation. Python: Production-quality development (modular code testing logging integration with APIs/files CICD Unit Test/Integration test automation Code Coverage).
• PySpark: Distributed transformations and performance optimization (joins partitions debugging) CICD Unit Test/Integration test automation Code Coverage.
• Azure Data Factory (ADF): Build/operate ADF pipelines parameterization triggers monitoring retry/error handling; integrate with Databricks/ADLS.
• Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration.
• Azure Fundamentals Pulumi: Hands-on with ADLS Gen2 Azure Portal Storage Explorer Resource Groups Azure SQL and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments
• Local to Chicago IL required as this needs candidates to go for in person client interview and 3 days Work from Office per Week.

Nice to have skills: -

• Ability to support/translate validation rules with SQL scripts and create data quality reports. TypeScript: Useful for pulumi pipeline to create Azure components.
• Java: Useful for integration with existing services/components.
• .NET: Useful for integration with existing services/components. Angular / Spring Boot: Minor troubleshooting or coordination with app teams.

Domain: Clinical (Health care)