Technical Lead Data Engineer

Data Engineering

We are seeking a highly skilled and experienced Data Engineeringwith strong expertise in Microsoft Azure data services enterprise data integration and modern data lake architectures. In this role you will lead the design development and optimization of scalable data pipelines responsible for ingesting and transforming data from enterprise systems including Microsoft platforms SAP CRM systems retail systems and external third-party data sources into a centralized data platform to support downstream applications analytics AI and agentic use cases.

You will play a critical role in building a robust high-performance data platform using Azure-native services ensuring data quality reliability and near real-time availability for business operations. The ideal candidate will have hands-on experience building Medallion Architecture (Bronze Silver Gold) data lakes integrating diverse enterprise data sources and enabling AI-powered solutions using the Microsoft ecosystem.

Responsibilities

Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems including Microsoft applications SAP CRM platforms retail systems and external third-party sources into Azure-based data platforms.

Design build and maintain enterprise-scale Medallion Architecture Data Lakes (Bronze Silver Gold) using Azure Data Lake Storage Microsoft Fabric Databricks or similar technologies.

Develop and optimize data processing workflows using Azure Databricks Apache Spark / PySpark Azure Data Factory Microsoft Fabric and other Azure-native services.

Create denormalized API-ready analytics-ready and AI-ready data models aligned with downstream applications microservices reporting and agentic AI consumption patterns.

Implement idempotent processing CDC merge (upsert) strategies incremental processing and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.

Leverage Azure Data Lake Storage Gen2 Azure SQL Azure Synapse Analytics Microsoft Fabric Cosmos DB and related services for efficient storage and querying of structured and semi-structured data.

Implement data ingestion patterns (batch streaming and event-driven) using tools such as Azure Event Hubs Azure Functions Logic Apps Service Bus and Azure Data Factory.

Lead enterprise data integration initiatives across Microsoft platforms SAP SaaS applications APIs partner systems and external data providers.

Apply performance tuning and optimization techniques to improve pipeline efficiency scalability reliability and cost-effectiveness.

Define and enforce data governance data quality metadata management and data lineage standards across the data platform.

Collaborate with DevOps teams to design and maintain CI/CD pipelines using Azure DevOps GitHub Actions or similar tools.

Support AI and agent-enabled solutions leveraging Azure AI Foundry Azure OpenAI Service Microsoft Copilot Studio and related Microsoft AI technologies.

Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.

Collaborate with application microservices analytics and AI teams to ensure seamless integration with enterprise data platforms via APIs event streams and data products.

Requirements

5 years of experience in data engineering with at least 2 years in a lead role.

Strong hands-on experience with Azure data services such as Azure Data Factory Azure Data Lake Storage Gen2 Azure Databricks Azure Synapse Analytics Microsoft Fabric and Azure SQL.

Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze Silver Gold) and modern lakehouse solutions.

Proficiency in PySpark / Spark for large-scale data processing and optimization.

Experience designing and implementing ODS Lakehouse or operational data layers using technologies such as Cosmos DB Azure SQL Microsoft Fabric Databricks or similar platforms.

Strong expertise in ETL/ELT design patterns enterprise data integration data ingestion and transformation pipelines.

Experience integrating data from Microsoft platforms SAP systems CRM applications REST APIs SaaS applications and external third-party data sources.

Hands-on experience with CI/CD pipelines (Azure DevOps GitHub Actions or similar).

Good understanding of data governance data quality metadata management and data lineage frameworks.

Experience supporting microservices analytics platforms AI solutions and agentic architectures with enterprise data platforms.

Exposure to Azure AI Foundry Azure OpenAI Service Microsoft Copilot Studio or related Microsoft AI technologies is highly desirable.

Strong problem-solving skills and ability to optimize complex data workflows.

Excellent communication and stakeholder management skills.

Ability to work in a fast-paced agile environment and manage multiple priorities.