Senior Lead Data & AI Engineer

Join us in a high-impact high-visibility role where you will pioneer world-class Data & AI engineering solutions building the next generation of intelligent agent-driven systems that power real-time business decisions.
We are seeking an expert Lead Data & AI Engineer with 9 to 12 years of experience to architect and deploy AI agents into business workflows focusing on Databricks and AWS environments. You will lead data engineering AI agent orchestration and scalable production-grade AI architectures.

Key Responsibilities:

• Architect and build reusable metadata-driven data and AI engineering frameworks that standardize ingestion transformation feature engineering and AI workflow deployment. Leverage Databricks lakehouse architecture declarative pipelines and cloud-native services to enable scalable governed and reusable data products across the organization.

• Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing AI pipelines and agent data refresh cycles.

• Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows.

• Integrate Lakeflow for real-time data stream processing ensuring AI agents are updated and responsive to live data.

• Ensure seamless orchestration between AI models and data pipelines using event-driven architectures for real-time inference and deployment.

• Implement and orchestrate AI agents using frameworks such as Agentic systems AgentOps tooling and solutions like Agents on Databricks (Agent-bricks).

• Hands-on experience deploying AI agents using RAG Graph RAG MCP-enabled integrations and agent orchestration frameworks such as AgentOps AgentBricks LangGraph or cloud-native orchestration services.

• Manage AI agent lifecycles monitoring and scaling using tools like SageMaker Bedrock or AI orchestration frameworks on AWS.

• Ensure robust data governance metadata management and AI observability through Unity Catalog AWS Glue or custom metadata layers.

• Design for scalability and modularity ensuring AI agents are reusable across multiple business processes.

Qualifications:

• 912 years in data engineering specializing in AI deployment within cloud ecosystems (Databricks AWS).

• Hands-on experience deploying AI agents using frameworks like RAG graph RAG and orchestrating agents (AgentOps Agent-bricks etc.).

• Proficient in AWS AI/ML services (SageMaker Bedrock) and orchestration tools (MWAA Step Functions).

• Strong knowledge of lakehouse architecture Unity Catalog and data modeling best practices.

• Deep experience in data orchestration monitoring and scalable AI-driven workflows.

How We Work

Vanguard has implemented a hybrid working model for the majority of our crew members designed to capture the benefits of enhanced flexibility while enabling in-person learning collaboration and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.