DataAI Engineer

Overview:

We are looking for a Senior Data Engineer with deep expertise in Lakehouse architecture real-time data streaming cloud data infrastructure and microservices development on Azure Kubernetes Service (AKS). You will play a central role in designing and delivering next-generation data pipelines BI solutions AI/ML platforms streaming APIs and scalable microservices that power Guidepoints research and analytics products.

This is a high-impact hands-on engineering role. You will work closely with data architects data scientists analysts frontend engineers QA and DevOps teams to translate complex business requirements into scalable reliable and observable data systems.

This is a Hybrid role from our Pune office.

What Youll Do:

Data Engineering & Lakehouse

• Design build and maintain ETL pipelines data ingestion workflows and table schemas on Azure Databricks to support BI analytics and AI/ML use cases

• Architect and optimize the Lakehouse using Delta Lake on Databricks ensuring reliability performance and cost efficiency

• Build and support data pipelines from business applications such as Salesforce NetSuite and other enterprise systems

• Develop and maintain Knowledge Graph models entity relationship structures and NLP-based insight pipelines

• Maintain data governance data privacy standards and compliance best practices throughout the data lifecycle

• Perform root cause analysis on data and processes to identify opportunities for improvement

• Collaborate with data architects scientists and business consumers to populate and optimize the data warehouse for reporting and analytics

Microservices & AKS Development

• Develop and support scalable web APIs and microservices using Python and Azure Platform Services

• Build new applications services and platforms; optimize existing solutions and refactor legacy components using modern scalable architectures

• Design implement and deploy microservices on Azure Kubernetes Service (AKS) using Docker Kubernetes Helm and Azure DevOps YAML pipelines

• Perform end-to-end deployments including infrastructure setup configuration and monitoring on AKS

• Decompose portions of legacy applications into modern microservices architecture

• Design and manage JSON payloads and payload contexts for inter-service communication

• Engage in database schema design and management including updating tables and rows for large datasets

• Collaborate with cross-functional teams Full-Stack QA DevOps and Product in agile SDLC processes

Real-Time Streaming & SSE

• Design and implement robust SSE (Server-Sent Events) endpoints using Python frameworks (FastAPI Flask Django) for real-time event delivery to web and mobile clients

• Build and maintain asynchronous backend services using asyncio aiohttp or similar libraries for non-blocking high-concurrency streaming

• Architect streaming data pipelines integrating SSE with upstream message brokers Kafka Redis Pub/Sub RabbitMQ

• Optimize connection lifecycle management: reconnection logic heartbeat signals event ID tracking and graceful shutdowns

• Collaborate with frontend teams to define and evolve SSE event schemas and API contracts

• Implement observability across streaming services: distributed tracing structured logging and metrics using Prometheus Datadog or OpenTelemetry

Engineering Excellence

• Write comprehensive unit integration and load tests for all data streaming and microservices components

• Write and maintain robust CI/CD pipelines using Azure DevOps YAML pipelines

• Participate in architecture reviews code reviews and on-call rotations

• Maintain thorough technical documentation and mentor junior engineers on best practices in data engineering Lakehouse architecture streaming systems and microservices

What You Have:

• Bachelors degree in Computer Science Engineering or a related field from an accredited university

• 7 years of professional data engineering and/or backend software engineering experience

• Advanced SQL expertise across relational and NoSQL databases (SQL Server Neo4j Elasticsearch Cosmos DB)

• Strong hands-on experience building and optimizing data pipelines on Azure Databricks

• In-depth knowledge of Delta Lake Data Warehousing and Lakehouse architecture

• Highly proficient in Spark Python and SQL

• Proven experience designing and deploying microservices on AKS using Docker Kubernetes and Helm

• Hands-on experience with Azure DevOps YAML pipelines for CI/CD automation

• Experience with SSE or real-time streaming event stream formatting retry logic connection management

• Strong grasp of async Python: asyncio async/await event loops

• Experience with message brokers: Kafka Redis Streams RabbitMQ or similar

• Proven track record of processing and extracting value from large complex and disconnected datasets

• Excellent stakeholder management and communication skills across global cross-functional teams

• Proven leadership skills with a strategic mindset and passion for driving innovation

Nice to Have

• Experience with Fivetran for data integration

• Familiarity with BI tools such as Power BI

• Experience building and deploying ML and feature engineering pipelines using MLflow

• Knowledge of Knowledge Graph development (e.g. Neo4j) and NLP-based analytics

• Familiarity with cloud-based AI/ML services and Generative AI tools

• Experience working in a compliance-based environment (building and deploying compliant software throughout the SDLC)

• Familiarity with API gateway configuration for streaming (NGINX Kong Azure API Gateway)

What We Offer:

• Competitive compensation
• Employee medical coverage
• Central office location
• Entrepreneurial environment autonomy and fast decisions
• Casual work environment

About Guidepoint:

Backed by a network ofnearly 1.75million experts andGuidepoints1600 employees worldwide we inform leading organizations research by delivering on-demand intelligence and research on request. With Guidepoint companies and investors can better navigate the abundance of information available today making it both more useful and more powerful.

At Guidepoint our success relies on the diversity of our employees advisors and client base which allows us to create connections that offer a wealth of perspectives. We are committed to upholding policies that contribute toan equitableand welcoming environment for our community regardless of background identity or experience.

#LI-AD2

#LI-HYBRID