Principal Software Engineer Jobs in Nashville, TN

About the team

We are a software development team building OCI services to operate AI infrastructure at scale. We transform complex operational processes into reliable AI-driven services, automated workflows, and AI agents that help engineers assess GPU infrastructure health, investigate and resolve issues, and make informed decisions.

Our work spans infrastructure lifecycle management, operational visibility, GPU infrastructure delivery, and AI-assisted automation. We partner with operations, infrastructure, and engineering teams to reduce repetitive work and improve the reliability and efficiency of large-scale GPU environments.

About the role

We are looking for a Principal Software Engineer to lead the design and delivery of software platforms for AI infrastructure. You will translate ambiguous problems into clear technical designs, build production services, and guide engineering decisions across multiple systems.

This is a hands-on individual contributor role. You will write and review code, own significant technical initiatives, mentor engineers, and remain accountable for how your systems perform in production.

What you will do

• Own architecture and delivery of services and automation supporting AI infrastructure provisioning, validation, troubleshooting, and ongoing operations.
• Build reliable APIs, backend services, integrations, and tools that connect infrastructure systems and turn data into actionable workflows.
• Develop AI-assisted capabilities for investigation, knowledge retrieval, and workflow execution, with appropriate evaluation, access controls, and human oversight.
• Replace recurring manual tasks with reusable automation, using operational feedback to prioritize improvements.
• Design for partial failures, retries, idempotency, auditability, observability, and safe recovery.
• Lead technical initiatives from problem definition through implementation, rollout, and production support.
• Establish engineering standards through design reviews, code reviews, testing, and clear technical documentation.
• Mentor engineers and align technical decisions with partner teams.
• Measure results through reduced manual effort, faster resolution, improved reliability, and adoption by operational users.

Required qualifications

• 7+ years of professional software engineering experience, including ownership of complex production systems.
• Strong programming skills in Python, Java, Go, or a comparable language.
• Experience designing and operating distributed systems, backend services, and APIs.
• Strong understanding of data modeling, SQL, asynchronous processing, and integration across multiple systems.
• Experience with Linux, containers, Kubernetes, and cloud infrastructure.
• Demonstrated ability to diagnose production issues involving application behavior, infrastructure, and data dependencies.
• Experience delivering secure, maintainable software with automated testing, deployment pipelines, monitoring, and operational documentation.
• Ability to lead technical work across teams, make sound architectural tradeoffs, and deliver under ambiguous requirements.
• Clear written and verbal communication skills, with experience mentoring engineers.
• Bachelor’s degree in computer science, engineering, or a related field, or equivalent practical experience.

Preferred qualifications

• Experience with GPU infrastructure, AI/ML platforms, HPC, or large-scale data center environments.
• Experience building infrastructure automation, orchestration platforms, or event-driven workflows.
• Experience delivering LLM applications or AI agents, including tool integration, retrieval, evaluation, and production monitoring.
• Familiarity with infrastructure inventory, telemetry, incident management, or hardware lifecycle systems.
• Experience with OCI or another major cloud platform.

• Lead the architecture, design, and development of OCI services that support AI infrastructure operations at scale.
• Build AI-driven services, automated workflows, and AI agents that help engineers assess GPU infrastructure health, investigate issues, and take corrective action.
• Design reliable distributed systems and APIs that integrate infrastructure telemetry, inventory, diagnostics, and operational data.
• Translate complex operational needs into clear technical requirements and deliver solutions from concept through production.
• Develop safeguards for AI-driven actions, including access controls, evaluation, auditability, and human oversight.
• Own service reliability, performance, scalability, and security through testing, observability, safe deployments, and production support.
• Partner with operations and infrastructure teams to identify recurring problems and replace manual processes with reusable automation.
• Set technical direction, lead design and code reviews, and mentor engineers to strengthen engineering practices.
• Measure and improve outcomes through reduced manual effort, faster issue resolution, and increased infrastructure availability.