Staff AI Engineer AI Labs

Technology Scouting & Evaluation

• Run short instrumented spikes and benchmarking on new models tools and frameworks: LLMs agentic systems vector databases orchestration frameworks copilots assistants and more.

• Compare vendor and open-source options documenting trade-offs across quality cost latency security and integration complexity.

• Deliver concise decision memos with clear recommendations: adopt watch or avoid.

Evaluation Harnesses & Sandboxes

• Design and maintain evaluation environments (e.g. datasets prompts scenarios telemetry) to test models under realistic constraints.

• Build automation and tooling to measure quality robustness latency and cost including regression tracking over time.

• Ensure every evaluated technology has benchmark coverage and a documented risk and limitations view.

Prototyping & Technical Validation

• Build enough of a system to understand how a technology behaves under realistic conditions not just in vendor demos or isolated examples.

• Explore architecture integration patterns operational constraints security boundaries and failure modes through working prototypes.

• Determine what must be true for a proof of concept to become a viable production capability.

• Prefer focused prototypes that answer specific technical questions over prematurely building production systems.

Recommendations Readiness & Hand-offs

• Translate technical findings into clear decision memos for both technical and non-technical stakeholders.

• For validated technologies produce readiness guidance covering recommended patterns guardrails known limitations operational considerations and integration requirements.

• Coordinate hand-offs to the engineering teams responsible for productionization.

• Support those teams during the transition when deep context from the evaluation is required without becoming the permanent owner of the resulting system.

• Track what happens after Lab recommendations and use those outcomes to improve future evaluation methods.

Governance Risk & Standards

• Work with Security Legal Compliance and other AI teams to document risk assessments mitigations and governance recommendations for each evaluated technology.

• Maintain checklists decision templates and lightweight standards reusable across evaluations and by partner teams.

• Incorporate learnings from third-party AI tooling already in use such as external copilots and the AWS AI suite into adoption guidelines.

Collaboration Mentoring & Community

• Partner with other AI teams and domain teams to ensure clear boundaries and smooth collaboration.

• Participate in hiring as a technical evaluator and culture champion.

• Mentor engineers in the Lab and adjacent teams on evaluation methods benchmarking and experimental design.

• Share knowledge through internal write-ups tech talks and occasional external meetups and conferences.

Technical depth

• 8 years of software engineering experience including significant experience operating at senior or Staff-level scope.

• Deep hands-on experience building and evaluating systems based on LLMs and modern AI tooling.

• Strong software engineering fundamentals and the ability to rapidly build high-quality experimental systems.

• Experience building agentic or multi-step AI systems involving tool use orchestration state retrieval or external integrations.

• Strong knowledge of cloud infrastructure preferably AWS and the ability to run experimental workloads securely and cost-consciously.

• Experience with observability telemetry testing and benchmarking of complex systems.

• Ability to reason about system architecture reliability scalability asynchronous workflows and distributed components where relevant.

• Track record of designing experiments or benchmarks that influenced meaningful technical decisions.

Benchmarking & evaluation

• Track record designing and running benchmarks that compare AI models and tools under real constraints.

• Experience constructing evaluation datasets: task selection labelling holdout discipline and keeping a set useful as models improve.

• Working knowledge of LLM-as-judge methods and their failure modes alongside human evaluation inter-annotator agreement and a view on when each is appropriate.

• Able to reason about statistical significance on small samples and to state confidence honestly rather than over-reading a result.

• Familiarity with regression tracking telemetry and versioning so that a result stays reproducible months later.

Decision-making

• Able to turn ambiguous we should try this new thing ideas into well-scoped evaluation plans with clear hypotheses and metrics.

• Comfortable making trade-off calls across quality latency cost and vendor lock-in and documenting them clearly.

• Experience writing short opinionated decision memos that help others move fast.

Collaboration & communication

• Can explain technical results to non-specialists in concrete concise terms.

• Experience working with platform product and operations teams to align evaluations with real use cases.

• Able to influence without authority aligning teams around shared standards and guardrails.

Mindset

• Curious and biased toward experimentation combined with disciplined measurement and risk awareness.

• Comfortable in a small high-leverage team without embedded PMs. You structure your own work and keep stakeholders informed.

• Builder attitude: you prefer reusable tools templates and playbooks over one-off work.