Senior AI Software Engineer Jobs in Virginia

MANTECH seeks a motivated, career and customer-oriented Senior AI ML Engineer. This is currently a hybrid position with two to three days onsite in Ashburn, VA.

In this role, you will collaborate within a cross-functional team to develop new Artificial Intelligence/Machine Learning (AI/ML) based solutions into operational pipelines to deliver mission impact for U.S. Customs and Border Protection (CBP). The ideal candidate will have deep expertise and experience with predictive modeling lifecycles, hands-on experience with machine learning tools and frameworks, and a pragmatic, customer-centric approach to applying ML models to solve complex problems.

Each day CBP oversees the massive flow of people, capital, and products that enter and depart the United States via air, land, sea, and cyberspace. The volume and complexity of both physical and virtual border crossings require the application of solutions to aid officers in detecting threats while promoting efficient trade and travel.

Responsibilities include but are not limited to:

• Lead the integration and deployment of trained AI/ML models into production environments (e.g., cloud, edge devices) using MLOps best practices.
• Develop and optimize model training & inference pipelines for real-time execution and efficiently handle large-scale data processing.
• Work with data science teams to structure automated ML model health monitoring and refresh capabilities.
• Implement continuous integration, delivery and training (CI/CD/CT) workflows with commercial and open-source modeling platforms/services.
• Coordinate with Data Science and Engineering teams to build scalable feature stores for optimal model training & execution workflows.
• Research, evaluate and recommend new tools, applications, software packages for MLOps engineering that can be adopted and approved for use in the CBP environment.
• Collaborate with cross-functional teams (e.g., Software Engineering, Data Science) to integrate and test multiple candidate AI/ML models and applications for operational assessment.

Required Qualifications:

• HS Diploma/GED and 15-20 years of experience, AS/AA and 13-18 years, BS/BA and 7+ years or MS/MA/MBA and 5+ years or PhD/Doctorate and 3+ years.
• Hands-on experience with LLMs such as Gemini, Llama, Mistral, or other open-source and commercial models. Experience with LLM application frameworks such as LangChain, LlamaIndex, or equivalent custom frameworks. Ability to optimize LLM systems for latency, throughput, scalability, reliability, GPU utilization, and inference cost. Experience deploying machine learning or LLM services in AWS, Azure, or Google Cloud. Demonstrated experience designing and deploying LLM solutions, including the following:
  • Retrieval-augmented generation (RAG)
  • Agentic workflows and tool calling
  • Prompt engineering and structured outputs
  • Model fine-tuning, e.g. LoRA
  • Embedding-based search and semantic retrieval
• Strong understanding of transformer architectures, tokenization, embeddings, context windows, inference parameters, and common LLM failure modes. Experience evaluating LLM applications for accuracy, relevance, hallucination, safety, latency, and cost.
• Experience with vector databases or search technologies such as OpenSearch, Elasticsearch, Milvus, Qdrant, Pinecone, Weaviate, or pgvector.
• Experience designing and integrating RESTful APIs and microservices using frameworks such as FastAPI.
• Working knowledge of SQL and experience with relational, document, or NoSQL databases.
• Familiarity with Docker, Kubernetes, CI/CD pipelines, monitoring, logging, and production incident troubleshooting.

Preferred Qualifications

• Master’s degree or Ph.D. in Computer Science, Machine Learning, Natural Language Processing, or a related discipline.
• Experience training, fine-tuning, quantizing, or serving open-source LLMs using tools such as PyTorch, Ollama, or TensorRT-LLM.
• Understanding of AI security risks, including prompt injection, data leakage, unsafe tool execution, model abuse, and adversarial inputs. Experience building multi-agent systems, multimodal applications, long-context workflows, or human-in-the-loop AI systems.
• Knowledge of advanced retrieval techniques, including hybrid search, reranking, query expansion, metadata filtering, and retrieval evaluation.
• Experience in LLM projects from initial requirements and proof of concept through production deployment and ongoing optimization.
• Strong knowledge of software engineering practices, including version control, code review, automated testing, system design, and technical documentation.

Clearance Requirements:

• Must be a U.S. Citizen and be able to obtain and maintain a CBP suitability prior to start this position.
• The ability to obtain and maintain a Top-Secret clearance.

Physical Requirements:

• The person in this position needs to occasionally move about inside the office to access file cabinets, office machinery, or to communicate with co-workers, management, and customers, which may involve delivering presentations,