Senior AI Software Engineer Jobs in Santa Clara, CA

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Research Superclusters (MARS), builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you'll help design solutions that power some of the world's most advanced computing workloads.

The MARS Scheduling team decides how NVIDIA's AI research compute gets used. That fleet spans many of the largest GPU clusters in the world, across multiple clouds and partners, running everything from frontier-model pretraining to reinforcement learning and agentic workloads. We run Slurm and Kubernetes at large scale. Our scheduling is coordinated and topology-sensitive on rack-scale systems. We manage fair sharing and quotas for hundreds of research teams. We support preemption and reservations. Elastic jobs recover from failure without losing progress. You will own real parts of the scheduling stack, and the opportunity to ship code into our open-source and internal platform code, and see the result in GPU-hours returned to researchers within days. The researchers training NVIDIA's own models are your users, and they will tell you quickly what works.

What you'll be doing:

• Design and build scheduling for large GPU fleets on both Slurm and Kubernetes. That covers queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations and backfill that reaches the end of very deep queues.
• Build the policy layer: fair share, quotas and allocations across many research teams, driven from a central allocation system and enforced the same way on every scheduler.
• Run Slurm as a cloud-native service on Kubernetes, and extend Kubernetes batch scheduling (Kueue and its ecosystem) for training, reinforcement learning, sandboxes and inference.
• Build services around the scheduler: reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs that let other platforms submit and track work.
• Measure and raise goodput: GPU occupancy, waste, fairness and time to start for large jobs. Use the data to choose what to build next.
• Own reliability for the components you build: on-call, incident analysis and fixes that remove whole classes of failure, so problems are found before researchers hit them.
Work upstream and across teams: contribute to Slurm, Kubernetes and related open-source projects, and partner with platform, storage, networking and research teams.

• 
  

What we need to see:

• BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.
• 6+ years building production systems software.
• Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn or a scheduler you built. You should be able to explain its internals, not only its configuration.
• Strong programming in Go, C or C++, plus Python for tooling and analysis.
• Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.
• Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.
• A track record of owning a service in production and making it measurably better.
Clear written and spoken communication with engineers and researchers.

• 
  

Ways to stand out from the crowd:

• Experience scheduling large-scale AI training: thousands of GPUs, NVLink and InfiniBand or RoCE topology, multi-node jobs that must start together.
• Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.
• Experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.
• Familiarity with RL, agentic or inference workloads and what they need from a scheduler: heterogeneous jobs, sandboxes, elastic scale and fast restarts.
• Experience with workflow orchestrators such as Flyte, Airflow, Ray or Kubeflow at scale.

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are:
184,000 USD – 287,500 USD for Level 4; and 224,000 USD – 356,500 USD for Level 5.

Applications for this job will be accepted at least until November 14, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.