TPU Performance Engineer

Role Overview

We seek a TPU Performance Engineer to enhance the efficiency of large-scale large language model (LLM) inference workloads on Google TPU platforms. This role involves deep collaboration with TPU hardware, compiler teams, and runtime developers to boost latency, throughput, and overall system performance.

Key Responsibilities

• Optimize inference processes for LLMs running on Google TPU hardware.
• Develop and refine the TPU backend infrastructure, including kernels, compiler integration, and runtime components.
• Improve performance-critical operations such as Attention mechanisms, General Matrix Multiply (GEMM), Key-Value caching, Sampling methods, and fused kernel implementations.
• Work extensively with JAX, XLA, Pallas, and related compiler/runtime technologies.
• Create benchmarking and profiling tools to identify bottlenecks across computation, memory usage, compilation, and runtime performance.
• Collaborate closely with teams focused on models, inference engines, compilers, and hardware to enhance deployment performance.

Candidate Requirements

• Bachelor's degree or equivalent expertise in fields like Computer Science, Engineering, Machine Learning, or Systems.
• Practical experience tuning TPU performance; familiarity with JAX, XLA, Pallas, or similar tools is preferred.
• Thorough knowledge of TPU architecture, including memory handling, compilation processes, and machine learning workload behaviors.
• Background in optimizing ML kernels, working on LLM inference, backend/runtime system development, or performance engineering.
• Proficient in programming with C++ and/or Python.
• Strong competencies with performance profiling and benchmarking methodologies.

Desirable Qualifications

• Experience with advanced LLM inference frameworks like vLLM, SGLang, TensorRT-LLM.
• Understanding of LLM serving architectures, including batching strategies, KV caching, decoding techniques, and methods for inference acceleration.
• Working knowledge of compiler toolchains such as MLIR, LLVM, Pallas, and XLA.
• Familiarity with numerical precision approaches like FP8, INT8, mixed precision, or quantization strategies.
• Contributions to open source AI infrastructure projects including vLLM, JAX/XLA, Pallas, or PyTorch/XLA.