AI Quality Analyst Jobs
AI Quality Analyst jobs are open across tech, healthcare, financial services, and enterprise software, at every level from associate to principal and director, with specializations in model evaluation, bias testing, and LLM output validation. Find a role that fits below and apply directly.
Find AI Quality Analyst JobsLooking for remote work? View remote AI quality analyst jobs →Overview
Showing 5 of 302+ AI Quality Analyst jobs
Karman Space & Defense is a leader in the rapid design, development, and production of critical, next-generation system solutions that align with the U.S. Department of War and its allies' core mission priorities, and meet the accelerating demand for access to space. Building on nearly 50 years of success, we deliver Payload & Protection Systems, Aero/Hydrodynamic Interstage Systems, and Propulsion & Launch Systems to more than 80 prime contractors supporting over 130 space and defense programs.
In this senior individual‑contributor role, you will translate Karman's enterprise artificial intelligence (AI) strategy into secure, scalable, and reusable architectures that accelerate responsible adoption across the organization. You will shape technical patterns, build reference implementations, evaluate emerging technologies, and establish guardrails that enable governed self‑service. Partnering closely with AI/Data & Automation, Business Systems, Cybersecurity, Infrastructure, and functional teams, you will provide hands‑on technical leadership that strengthens solution quality, operational consistency, and enterprise readiness.
Responsibilities
- Defines and evolves Karman's enterprise AI reference architecture.
- Establishes architectural patterns for model access, retrieval‑augmented generation (RAG), orchestration, agents, application programming interfaces (APIs), evaluation, observability, and cost management.
- Develops technical standards for Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry, Azure OpenAI, Microsoft Fabric, and other approved enterprise AI capabilities.
- Evaluates external platforms and models based on security, performance, cost, and business needs.
- Designs architectures that support portability using APIs, abstraction layers, open protocols, and exit strategies.
- Builds proofs of concept, reference implementations, reusable components, and technical accelerators.
- Provides hands‑on support for complex AI use cases, including RAG solutions, agentic workflows, model‑integrated applications, and automation.
- Establishes reusable templates, patterns, guardrails, and documentation to enable governed self‑service.
- Translates enterprise AI governance requirements into technical controls for identity, access, logging, monitoring, classification, retention, and human oversight.
- Defines evaluation, observability, quality, operational readiness, incident response, and lifecycle management standards for AI solutions.
Required Qualifications
- Significant experience in enterprise architecture, cloud architecture, software engineering, or platform engineering.
- Recent experience designing and deploying production generative artificial intelligence (AI) solutions.
- Strong working knowledge of the Microsoft cloud and AI ecosystem, including Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry, Azure OpenAI, Microsoft Fabric, Entra ID, Purview, and Microsoft security technologies.
- Hands‑on experience integrating commercial AI models through APIs and building model‑enabled applications.
- Strong understanding of RAG architectures, agent patterns, tool use, model orchestration, vector search, and enterprise data integration.
- Familiarity with standards and technologies such as Model Context Protocol (MCP), OpenTelemetry, open‑weight models, and modern inference platforms.
- Experience designing secure systems in regulated or high‑assurance environments.
- Ability and willingness to write code, build prototypes, review designs, and troubleshoot technical issues.
- Strong written and verbal communication skills with the ability to explain complex technical tradeoffs to technical and executive audiences.
Preferred Qualifications
- Experience in aerospace, defense, government, manufacturing, financial services, or other regulated industries.
- Experience with GCC High, sovereign cloud, restricted environments, or controlled‑data architectures.
- Familiarity with frameworks such as Cybersecurity Maturity Model Certification (CMMC), National Institute of Standards and Technology (NIST) Special Publication 800‑171, Controlled Unclassified Information (CUI), International Traffic in Arms Regulations (ITAR), NIST AI Risk Management Framework (RMF), and International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 42001.
- Experience integrating AI with enterprise platforms such as enterprise resource planning (ERP), product lifecycle management (PLM), manufacturing execution systems (MES), customer relationship management (CRM), productivity, analytics, and workflow systems.
- Familiarity with graphics processing unit (GPU) infrastructure, self‑hosted models, fine‑tuning, and private inference environments.
- Relevant Microsoft, cloud, architecture, or security certifications.
This position requires U.S. person status under U.S. export control laws, including U.S. citizens and nationals, lawful permanent residents, refugees, and asylees.
Benefits
- Medical, dental, and vision insurance
- 401(k) with company match
- Paid time off
- Health Savings Account (HSA) with company contribution
- Flexible Spending Accounts (FSA)
- Company‑paid life and ADdummyamp;D insurance
- Short‑ and long‑term disability coverage
- Tuition reimbursement
Karman Space and Defense is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, protected veteran status, or any other status protected by applicable federal, state, or local law.
View our Privacy Policy
Karman Space & Defense is a leader in the rapid design, development, and production of critical, next-generation system solutions that align with the U.S. Department of War and its allies' core mission priorities, and meet the accelerating demand for access to space. Building on nearly 50 years of success, we deliver Payload & Protection Systems, Aero/Hydrodynamic Interstage Systems, and Propulsion & Launch Systems to more than 80 prime contractors supporting over 130 space and defense programs.
In this senior individual‑contributor role, you will translate Karman's enterprise artificial intelligence (AI) strategy into secure, scalable, and reusable architectures that accelerate responsible adoption across the organization. You will shape technical patterns, build reference implementations, evaluate emerging technologies, and establish guardrails that enable governed self‑service. Partnering closely with AI/Data & Automation, Business Systems, Cybersecurity, Infrastructure, and functional teams, you will provide hands‑on technical leadership that strengthens solution quality, operational consistency, and enterprise readiness.
Responsibilities
- Defines and evolves Karman's enterprise AI reference architecture.
- Establishes architectural patterns for model access, retrieval‑augmented generation (RAG), orchestration, agents, application programming interfaces (APIs), evaluation, observability, and cost management.
- Develops technical standards for Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry, Azure OpenAI, Microsoft Fabric, and other approved enterprise AI capabilities.
- Evaluates external platforms and models based on security, performance, cost, and business needs.
- Designs architectures that support portability using APIs, abstraction layers, open protocols, and exit strategies.
- Builds proofs of concept, reference implementations, reusable components, and technical accelerators.
- Provides hands‑on support for complex AI use cases, including RAG solutions, agentic workflows, model‑integrated applications, and automation.
- Establishes reusable templates, patterns, guardrails, and documentation to enable governed self‑service.
- Translates enterprise AI governance requirements into technical controls for identity, access, logging, monitoring, classification, retention, and human oversight.
- Defines evaluation, observability, quality, operational readiness, incident response, and lifecycle management standards for AI solutions.
Required Qualifications
- Significant experience in enterprise architecture, cloud architecture, software engineering, or platform engineering.
- Recent experience designing and deploying production generative artificial intelligence (AI) solutions.
- Strong working knowledge of the Microsoft cloud and AI ecosystem, including Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry, Azure OpenAI, Microsoft Fabric, Entra ID, Purview, and Microsoft security technologies.
- Hands‑on experience integrating commercial AI models through APIs and building model‑enabled applications.
- Strong understanding of RAG architectures, agent patterns, tool use, model orchestration, vector search, and enterprise data integration.
- Familiarity with standards and technologies such as Model Context Protocol (MCP), OpenTelemetry, open‑weight models, and modern inference platforms.
- Experience designing secure systems in regulated or high‑assurance environments.
- Ability and willingness to write code, build prototypes, review designs, and troubleshoot technical issues.
- Strong written and verbal communication skills with the ability to explain complex technical tradeoffs to technical and executive audiences.
Preferred Qualifications
- Experience in aerospace, defense, government, manufacturing, financial services, or other regulated industries.
- Experience with GCC High, sovereign cloud, restricted environments, or controlled‑data architectures.
- Familiarity with frameworks such as Cybersecurity Maturity Model Certification (CMMC), National Institute of Standards and Technology (NIST) Special Publication 800‑171, Controlled Unclassified Information (CUI), International Traffic in Arms Regulations (ITAR), NIST AI Risk Management Framework (RMF), and International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 42001.
- Experience integrating AI with enterprise platforms such as enterprise resource planning (ERP), product lifecycle management (PLM), manufacturing execution systems (MES), customer relationship management (CRM), productivity, analytics, and workflow systems.
- Familiarity with graphics processing unit (GPU) infrastructure, self‑hosted models, fine‑tuning, and private inference environments.
- Relevant Microsoft, cloud, architecture, or security certifications.
This position requires U.S. person status under U.S. export control laws, including U.S. citizens and nationals, lawful permanent residents, refugees, and asylees.
Benefits
- Medical, dental, and vision insurance
- 401(k) with company match
- Paid time off
- Health Savings Account (HSA) with company contribution
- Flexible Spending Accounts (FSA)
- Company‑paid life and ADdummyamp;D insurance
- Short‑ and long‑term disability coverage
- Tuition reimbursement
Karman Space and Defense is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, protected veteran status, or any other status protected by applicable federal, state, or local law.
View our Privacy Policy
AI Quality Analyst Jobs by Experience Level
See All 302+ AI Quality Analyst Jobs
Find roles that match your experience and apply in just a few clicks.
Find AI Quality Analyst JobsAI Quality Analyst Job Market
Who's Hiring
- Booz Allen Hamilton20
- JPMorganChase11
- Johns Hopkins University Applied Physics Laboratory9
- University of San Diego6
- Penn State University4
Top Industries Hiring
- Technology & Software
- Education
- Banking & Financial Services
- Healthcare & Medical Services
- Insurance
What Employers Look For
The qualifications that appear most often in AI quality analyst jobs.
- Experience designing and executing evaluation frameworks for large language models or ML systems
- Proficiency with annotation platforms and data labeling quality workflows
- Familiarity with bias detection, fairness testing, and responsible AI principles
- Strong analytical skills with Python or SQL for querying evaluation datasets
- Experience writing detailed rubrics, style guides, or quality guidelines for annotation teams
- Bachelor's degree in computer science, linguistics, cognitive science, or a related field
Tips for Your AI Quality Analyst Job Search
Tailor your resume to evaluation frameworks
Hiring managers for ai quality analyst roles scan for hands-on experience with specific evaluation frameworks like ROUGE, BLEU, or human-in-the-loop annotation pipelines. Name the frameworks you've used and the model types you evaluated, not just 'AI testing.'
Highlight labeled dataset experience clearly
Many candidates bury annotation and data labeling work under generic QA experience. Treat it as a headline skill. Describe the data types you labeled, the guidelines you followed, and the disagreement resolution process you used on your team.
Apply early to roles that fit
Migrate Mate lists ai quality analyst openings from across the United States in one place, so you can find roles that match and apply directly to each listing.
Target postings that name the model stack
Job listings that specify GPT-4, Gemini, Claude, or open-source models like LLaMA signal a team actively shipping AI products. Prioritize these over generic 'AI quality' postings, and mirror that model vocabulary in your application materials.
Prepare a rubric sample for the interview
Interviewers frequently ask you to design or critique an evaluation rubric on the spot. Walk in with a mental template: criteria, scoring scale, edge-case handling, and inter-rater reliability approach. Candidates who structure rubrics fluently advance faster.
Negotiate scope alongside compensation
In offers for ai quality analyst roles, model access, annotation tool budget, and team size matter as much as title. During negotiation, ask what percentage of work is human evaluation versus automated testing so you can assess the role's growth trajectory accurately.
AI Quality Analyst Jobs: Frequently Asked Questions
Which companies are hiring the most ai quality analysts?
The most active employers for ai quality analysts right now are Booz Allen Hamilton, JPMorganChase, and Johns Hopkins University Applied Physics Laboratory, and the most openings are in Virginia, California, and Texas, based on current listings on Migrate Mate as of November 2026. Demand is highest at companies actively deploying generative AI products at scale.
How many ai quality analyst jobs are remote?
About 54% of ai quality analyst openings are fully remote or hybrid as of November 2026, reflecting the nature of evaluation and annotation work that rarely requires physical presence. Sub-areas like prompt evaluation, red-teaming, and LLM output review are especially concentrated in remote roles, since the work is conducted entirely through web-based tooling and asynchronous review cycles.
How do you become an ai quality analyst?
Start by building hands-on experience with model evaluation, which you can do through open-source annotation projects, academic research, or contributing to public benchmarks. Develop a working knowledge of evaluation metrics like ROUGE, BERTScore, or human preference scoring. Document rubrics you've created or quality processes you've improved. Many practitioners move into the role from QA engineering, computational linguistics, or data annotation management backgrounds.
How do you get hired as an ai quality analyst with little experience?
Focus on demonstrating structured thinking about output quality rather than production experience. Build a portfolio of evaluation rubrics you've designed for public model outputs, contribute to open annotation datasets, or complete a project comparing responses from two different language models using a defined scoring framework. Roles titled 'AI Rater,' 'Data Quality Reviewer,' or 'Annotation Specialist' are common entry points that build directly toward a full analyst position.
What does the ai quality analyst interview process look like?
Most processes include a recruiter screen, a technical assessment where you evaluate a set of model outputs using a provided rubric, and a panel interview with the AI or data team. The take-home assessment almost always involves rating or rewriting model responses and explaining your reasoning. Some teams add a live rubric design exercise where you define evaluation criteria for a novel task from scratch, testing your ability to handle ambiguous quality standards.
Where can I find and apply to ai quality analyst jobs?
You can find and apply to ai quality analyst jobs on Migrate Mate, which lists current openings from across the United States. Find roles that match your experience and specialization, then apply directly to each listing. New positions are added regularly, so checking back frequently gives you the best chance of catching openings early.
See All 302+ AI Quality Analyst Jobs
Find roles that match your experience and apply in just a few clicks.
Find AI Quality Analyst Jobs