Overview
Senior QA Automation Engineer Jobs in Abu Dhabi, UAE/Dubai at TALENTMATE
Headquartered in Abu Dhabi, the company specializes in developing defense‑grade artificial intelligence solutions that satisfy rigorous reliability and validation standards for mission‑critical applications.
Job Summary
We are seeking a Senior QA Automation Engineer to lead the validation and verification strategies for our AI transformation initiative. In this critical role, you will define and enforce rigorous standards for non‑deterministic AI systems, ensuring Large Language Models (LLMs) and predictive engines meet the stringent reliability requirements of the defense sector. Your expertise will bridge Agile development methodologies with formal Systems Engineering practices, creating automated testing frameworks that validate both software functionality and AI behaviors against established ground‑truth datasets.
Working within a structured Stage Gate delivery model, you will ensure AI agents successfully pass Test Readiness Reviews (TRR) and Functional Configuration Audits (FCA) before deployment. This role demands a deep understanding of AI evaluation techniques, including hallucination detection, consistency validation, and factual accuracy assessment, while also addressing integration challenges with enterprise systems such as SAP S/4
HANA and Ariba. Your work will directly contribute to the trustworthiness of AI‑driven decision‑making in mission‑critical applications.
Key Responsibilities
- Lead the validation and verification of AI systems by architecting automated testing frameworks to evaluate non‑deterministic outputs, including hallucination detection, consistency checks, and factual accuracy against gold‑standard datasets for LLMs and predictive engines.
- Design and implement automated evaluation metrics such as RAGAS, faithfulness, and answer relevance to validate Retrieval‑Augmented Generation pipelines, ensuring accurate citation of internal technical documentation and regulatory texts.
- Develop prompt regression suites to monitor prompt drift and maintain the quality of AI‑generated engineering documents, mitigating degradation risks from underlying model or system instruction changes.
- Construct robust integration tests to validate data consistency between AI agents and critical enterprise systems (e.g., SAP S/4
HANA, Ariba), ensuring no corruption of Bill of Materials or financial data in supply‑chain applications.
- Create performance benchmarking tests to validate latency and throughput of forecasting models and risk‑scoring engines, aligning with real‑time requirements for supply‑chain dashboards.
- Automate API validation for secure gateways, verifying role‑based access control (RBAC) and personally identifiable information (PII) redaction logic before data ingestion into AI models.
- Ensure alignment of automated test cases with system requirements and user needs, generating digital evidence for verification and validation reports as per Systems Engineering Handbook standards.
- Prepare test readiness packages for Stage Gate reviews, providing quantitative evidence to demonstrate system stability and readiness for progression from minimum viable product to production deployment.
- Manage the defect lifecycle between requirements quality assistants and development teams, tracing defects in AI logic back to specific model versions or datasets for resolution.
- Bridge Agile development and formal Systems Engineering processes, ensuring AI agents meet rigorous Test Readiness Reviews and Functional Configuration Audits for defense‑sector deployment.
Qualifications and Experience
- 5+ years of experience in QA automation, with at least 2 years focused on testing complex data‑driven applications, machine learning models, or AI agents.
- Experience in defense, aerospace, or highly regulated industries is a strong plus. Understanding of IV&V processes is highly desirable.
- Expert proficiency in Python for building custom test harnesses (Pytest) and standard automation libraries (Selenium/Playwright for UI, Requests for API).
- Expert proficiency in crafting performance test plans and implementations (Locust, JMeter, K6).
- Experience utilizing frameworks for evaluating large language models (Deep Eval, Tru Lens, or…
Title: Senior QA Automation Engineer
Company: TALENTMATE
Location: Abu Dhabi, UAE/Dubai
Category: