Senior AI Engineer

Job Title: Senior AI Engineer
Location: Remote, USA
Clearance Required: Active Secret Clearance
Position Type: Full-Time

About the Company

At VivSoft, we aim to solve complex federal problems using emerging and open technologies in a collaborative and rewarding environment. VivSoft is a diverse team of strategists, engineers, designers, and creators experienced in building high-performance, effective software, with a focus on impactful organizational design and software delivery dynamics. We build secure Software Factories based on DoD reference designs and NIST Frameworks for Cloud and DevSecOps. These factories deliver AI/ML Applications, Data Science Platforms, Blockchain, and Microservices for DoD, Healthcare, and Civilian Agencies.

Job Summary

We are seeking a Senior AI Engineer to design, develop, and deploy advanced Artificial Intelligence and Machine Learning (AI/ML) solutions within complex, large-scale data environments. This role will focus on building distributed machine learning models, developing agentic AI workflows, and integrating Large Language Models (LLMs) with enterprise data platforms to support predictive analytics, intelligent automation, and data-driven decision-making.

The ideal candidate will have extensive experience in software engineering, data engineering, and applied AI/ML, with expertise in distributed computing, statistical modeling, and modern AI technologies. This individual will work closely with data engineers, data scientists, and cloud/platform engineering teams to develop scalable AI solutions, optimize model performance, and deploy AI/ML capabilities within secure and disconnected environments.

Key Responsibilities

• Design, develop, and deploy distributed machine learning models using Apache Spark and Apache Iceberg, applying rigorous statistical validation and classical machine learning techniques where appropriate.
• Build and implement agentic AI workflows using LLMs, tool calling, Model Context Protocol (MCP), retrieval-augmented generation (RAG), and multi-step planning to query and analyze enterprise datasets.
• Develop and optimize AI-driven data analysis workflows that leverage distributed query engines such as Amazon Athena, Trino, and Spark SQL.
• Optimize agent-generated SQL queries through partition pruning, file layout awareness, query guardrails, and distributed query performance techniques.
• Design, deploy, and maintain self-hosted AI/ML model-serving environments using Kubernetes, including deployments within disconnected or restricted network environments.
• Develop and orchestrate automated model training, evaluation, and agent workflows using Apache Airflow, AWS services, and Kubernetes.
• Leverage AWS technologies, including Amazon S3, EC2, and EMR, to support distributed AI/ML processing and deployment.
• Evaluate LLM and AI agent outputs using structured test datasets, evaluation rubrics, regression testing, and performance metrics.
• Monitor AI/ML solutions for model drift, performance degradation, and operational failure modes.
• Collaborate with data engineers, data scientists, and cloud/platform engineers to integrate AI/ML models into production data pipelines and enterprise environments.
• Apply responsible AI principles, including bias assessment, model transparency, data handling, and risk management.
• Provide technical leadership, mentor junior engineers, and contribute to engineering best practices.
• Develop and maintain technical documentation, architecture designs, operational runbooks, and implementation guidance.
• Communicate model behavior, limitations, and technical risks to technical and nontechnical stakeholders.

Qualifications & Required Skills

• Active U.S. Government Secret security clearance.
• U.S. Citizenship required.
• Minimum of 8 years of professional experience in software engineering, data engineering, or machine learning, including at least 4 years of applied AI/ML experience.
• Bachelors degree in Computer Science, Information Systems, Engineering, Data Science, or a related technical field. Additional relevant professional experience may substitute for education on a year-for-year basis.
• Demonstrated experience developing and deploying distributed machine learning solutions using Apache Spark, including Spark MLlib, and lakehouse table formats such as Apache Iceberg.
• Strong statistical background, including probability, statistical inference, regression analysis, experimental design, and model evaluation.
• Experience applying classical machine learning algorithms and techniques to large-scale datasets.
• Hands-on experience designing and developing agentic AI workflows involving tool calling, MCP, and multi-step AI agents.
• Experience working with LLMs, retrieval-augmented generation (RAG), and vector databases.
• Strong understanding of distributed SQL query engines, including Amazon Athena, Trino, and Spark SQL, with the ability to optimize AI-generated queries.
• Experience deploying and serving AI/ML models in self-hosted or disconnected environments.
• Strong programming experience with Python.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Experience with AWS cloud services, including S3, EC2, and EMR.
• Experience deploying and managing applications or AI/ML workloads using Kubernetes.
• Strong analytical, critical-thinking, and problem-solving skills.
• Demonstrated ability to collaborate with multidisciplinary technical teams to deploy and operationalize AI/ML solutions.
• Ability to explain complex AI/ML concepts, model behavior, limitations, and technical risks to nontechnical stakeholders.
• Experience providing technical leadership and mentoring junior engineers.
• Understanding of responsible AI practices, including bias, model evaluation, and secure data handling.

Preferred Qualifications & Skills

• Masters degree or Ph.D. in Statistics, Computer Science, or a related quantitative field.
• Experience developing MCP servers or working with established AI agent frameworks.
• Experience implementing LLM evaluation frameworks, automated testing methodologies, and AI guardrails.
• Advanced knowledge of Apache Iceberg architecture and internals.
• Experience working with AWS Glue Data Catalog.
• Experience with Terraform and Infrastructure as Code (IaC).
• Prior experience supporting federal or DoD AI/ML programs.
• Experience designing and implementing scalable AI/ML solutions in secure, highly regulated environments.

Benefits

• Comprehensive Medical, Dental, and Vision Plans
• Life Insurance
• Paid Time Off (Flexible/Combined PTO, Bereavement Leave, 11 Company Paid Holidays)
• 401K Retirement Plan with employer match
• Professional Development Training Reimbursement