Technical Director, Data Engineering

Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms.

This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code debugging distributed systems evaluating frameworks and getting hands-on with architecture and implementation.

You will work across large-scale ingestion pipelines search systems AI/LLM infrastructure event-driven architectures vector databases analytics platforms and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job.

The ideal candidate has strong opinions informed by real-world experience understands tradeoffs deeply and can move quickly without creating unnecessary complexity.

• Design and build scalable data platforms and distributed processing systems

• Develop modern ingestion transformation and retrieval pipelines for structured and unstructured data

• Build systems supporting AI/LLM applications semantic search RAG pipelines vector search and agentic workflows

• Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware

• Evaluate and standardize frameworks tooling and infrastructure patterns across teams

• Improve performance reliability observability and cost efficiency of data systems

• Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities

• Drive pragmatic engineering decisions balancing speed maintainability and operational simplicity

• Mentor engineers technically through design reviews architecture guidance and pair debugging

• Help establish engineering standards around CI/CD testing data quality monitoring and operational excellence

Candidates should have significant real-world experience building and operating production systems using many of the following:

• Airflow

• Elasticsearch / OpenSearch

• Vector databases and semantic retrieval systems

• MongoDB PostgreSQL DynamoDB or similar platforms

• AWS

• AWS CDK

• Serverless architectures

• Distributed observability and monitoring stacks

• LLM integration patterns

• RAG architectures

• Embeddings and vector search

• MCP servers and AI orchestration frameworks

• LangChain LlamaIndex DSPy or similar ecosystems

• AI evaluation tracing and observability tooling

• Search relevance and ranking systems

• Python strongly preferred

• Experience with Java Go or TypeScript is a plus

• API design and distributed service architectures

• Event-driven and asynchronous systems

• Still enjoys building and debugging systems directly

• Has strong technical depth not just architectural vocabulary

• Comfortable operating in ambiguity and fast-moving environments

• Understands how to simplify systems instead of endlessly abstracting them

• Has experience modernizing legacy platforms and evolving architectures incrementally

• Can distinguish between engineering fundamentals and hype cycles

• Values shipping working systems over theoretical perfection

• Engineering teams can move faster because the underlying platforms are reliable and scalable

• AI and data systems become production-grade rather than experimental prototypes

• Infrastructure costs and operational complexity are reduced through better architecture

• Search ingestion and retrieval systems improve significantly in performance and relevance

• Teams adopt consistent maintainable technical patterns without excessive bureaucracy

• Experience with large-scale news document or regulatory data pipelines

• Experience with search relevancy tuning and semantic retrieval

• Exposure to governance compliance risk or enterprise SaaS platforms

• Familiarity with modern AI engineering workflows and agentic systems

• Experience supporting both startup-speed execution and enterprise-scale operations

About Us

Diligent is the AI leader in governance risk and compliance (GRC) SaaS solutions helping more than 1 million users and 700000 board members to clarify risk and elevate governance. The Diligent One Platform gives practitioners the C-Suite and the board a consolidated view of their entire GRC practice so they can more effectively manage risk build greater resilience and make better decisions faster.

Learn more or follow us onLinkedInandFacebook

What Diligent Offers You

• Creativity is ingrained in our culture. We are innovative collaborators by nature. We thrive in exploring how things can be differently both in our internal processes and to help our clients
• We care about our people.Diligent offers a flexible work environment global days of service comprehensive health benefits meeting free days generous time off policy and wellness programsto name a few
• We have teams all over the world. We may be headquartered in New York City but we have office hubs in Washington D.C. Vancouver London Galway Budapest Munich Bengaluru Singapore and Sydney.
• Diversity is important to us. Growing maintaining and promoting a diverse team is a top priority for us. We foster and encourage diversity through our Employee Resource Groups and provide access to resources and education to support the education of our team facilitate dialogue and foster understanding.

Diligent created the modern governance movement. Our world-changing idea is to empower leaders with the technology insights and connections they need to drive greater impact and accountability to lead with purpose. Our employees are passionate smart and creative people who not only want to help build the software company of the future but who want to make the world a more sustainable equitable and better place.

Headquartered in New York Diligent has offices in Washington D.C. London Galway Budapest Vancouver Bengaluru Munich Singapore and Sydney. To foster strong collaboration and connection this role will follow a hybrid work model. If you are within a commuting distance to one of our Diligent office locations you will be expected towork onsite at least 50% of the time.We believe that in-person engagement helps drive innovation teamwork and a strong sense of community.

_{We are a drug free workplace. Diligent is proud to be an equal opportunity employer. We do not discriminate based on race color religious creed sex national origin ancestry citizenship status pregnancy childbirth physical disability mental disability age military status protected veteran status marital status registered domestic partner or civil union status gender (including sex stereotyping and gender identity or expression) medical condition (including but not limited to cancer related or HIV/AIDS related) genetic information or sexual orientation in accordance with applicable federal state and local also consider qualified applicants regardless of criminal histories consistent with legal requirements. See alsoDiligents EEO Policy and Know Your are committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability you may contact us at .}

_{At Diligent we recognize the value of a diverse global workforce. Visa sponsorship may be available for select positions based on business needs and specific role requirements. Sponsorship decisions are evaluated on a case-by-case basis and eligibility should not be assumed for all opportunities.}

_{Please find the link to our candidate privacy policy here: all recruitment agencies: Diligent does not accept unsolicited agency resumes. Please do not forward resumes to our jobs alias Diligent employees or any other organization location. Diligent is not responsible for any fees related to unsolicited resumes.}