Principal Software Engineer AI Runtime & Distributed Inference Systems

About the Role

We are building the next generation of AI infrastructure: a large-scale inference platform powered

by custom AI accelerators capable of serving frontier dense and Mixture-of-Experts (MoE)

foundation models at datacenter scale. As a Principal Software Engineer you will define and lead

the architecture of the software stack that transforms hundreds to thousands of accelerator ASICs

into a unified inference system. You will drive the design of distributed runtimes communication

libraries scheduling systems memory management and model-serving infrastructure that deliver

industry-leading throughput latency and efficiency. This is a highly cross-functional leadership

role working across silicon architecture networking compiler runtime infrastructure and AI

research teams.

What Makes This Role Unique

This role sits at the intersection of distributed systems AI infrastructure networking compilers

and computer architecture. You will help shape the software foundation for a new class of AI

supercomputers and define how future foundation models are served efficiently at scale. The ideal

candidate is equally comfortable discussing transformer architectures network topology runtime

scheduling memory systems and accelerator microarchitecture and can translate those insights

into production-grade software that powers large-scale AI services.

Responsibilities

Technical Leadership

Define the software architecture for large-scale AI inference systems.

Lead the evolution of the distributed runtime stack for dense and MoE foundation models.

Establish long-term technical strategy for performance scalability reliability and

programmability.

Influence hardware-software co-design decisions across future accelerator generations.

Distributed Inference Runtime

Architect runtime systems supporting:

o Tensor parallelism

o Pipeline parallelism

o Expert parallelism

o Data parallelism

o Multi-rack inference execution

Design systems for efficient prefill and decode execution.

Drive continuous batching and request scheduling strategies.

Optimize TTFT TPOT throughput utilization and QoS. Experience in AI accelerator heterogenous programming models like CUDA/HIP/Triton

implementing device management memory management Kernel launching Error handling

and diagnostics Kernel optimization techniques with tiling and fused kernels.

MoE Infrastructure

Lead design of token routing and expert execution systems.

Define expert placement and replication strategies.

Develop mechanisms for expert load balancing and congestion control.

Optimize communication and compute overlap across distributed expert clusters.

Communication and Fabric Software

Architect high-performance collective communication libraries.

Design software interfaces for custom accelerator fabrics.

Develop scalable implementations of:

o AllReduce

o AllGather

o ReduceScatter

o Broadcast

o Expert dispatch and gather

Drive latency and bandwidth optimization across accelerator clusters.

Memory Systems

Define architecture for distributed KV-cache management.

Drive memory placement eviction and tiering policies.

Optimize utilization across local memory pooled memory and storage tiers.

Create runtime services for dynamic memory allocation and workload balancing.

Performance Engineering

Establish performance methodology benchmarks and KPIs.

Analyze bottlenecks across:

o Compute

o Memory bandwidth

o Communication fabric

o Expert routing

o Scheduling layers

Lead root-cause analysis of production performance and Production Systems

Drive reliability observability telemetry and debugging architecture.

Define health monitoring and fleet management capabilities.

Partner with infrastructure teams to operate inference systems at scale.

Establish production readiness standards and operational excellence practices.

System and Infrastructure experience - Experience implementing Virtualization

software Rack controllers Pcie/Memory controllers device life cycle management

software Experience with NCCL/RCCL/MPI collective communications Scale Up

networking with UALoE/ESUN/NVLink scale out networking with RoCEv2 Network

congestion control techniques like PFC/DCQCN.

Qualifications

Required

related field.

BS MS or PhD in Computer Science Computer Engineering Electrical Engineering or

12 years of software engineering experience.

5 years building distributed systems at scale.

Strong expertise in C Rust or similar systems programming languages.

Deep understanding of large-scale AI inference systems.

Experience with networking and distributed communication systems.

Proven track record leading architecture across multiple engineering teams.

Experience driving projects from concept through production deployment.

Preferred

Experience with large language model serving systems.

Experience with vLLM SGLang TensorRT-LLM Triton PyTorch Distributed NCCL or

equivalent technologies.

Experience designing runtime systems for custom accelerators GPUs TPUs or AI ASICs.

Knowledge of transformer architectures sparse models and MoE systems.

Experience with RDMA InfiniBand RoCE PCIe Ethernet fabrics or custom interconnects.

Experience in compiler technologies graph optimization or kernel scheduling.

Familiarity with HBM memory hierarchy design and cache management strategies.

Success Metrics

In the first 1224 months this leader will be expected to:

Deliver industry-leading TTFT and TPOT performance.

Achieve high accelerator utilization across production clusters. Enable scalable execution of frontier dense and MoE models.

Improve throughput per rack and throughput per watt.

Reduce communication overhead across distributed inference deployments.

Establish a robust and production-ready AI runtime platform.

Influence future hardware generations through software-driven architectural insights.