QA Engineer for AI Solutions

Middle QA Engineer for AI solutions at Kokoc Tech. Remote in Russia, long-term project. Up to 1400 RUB/hour gross. Support AI team on Dify platform.

The specialist is involved in supporting the AI team (RnD AI Team), which implements a portfolio of AI initiatives for business units on the internal Dify platform (LLM agents, RAG, classification, data extraction from documents, API/MCP integrations). The portfolio includes ~200 initiatives, with 10-15 in progress simultaneously. Tasks: 1. Preparing test cases based on requirements and acceptance criteria of initiatives; 2. Functional and integration testing of AI agents, workflows on the Dify platform, API integrations with internal systems; 3. Checking positive, negative, and edge scenarios, including atypical user requests and model 'hallucinations'; 4. Regression testing after changes to prompts, models, knowledge base (RAG), and integrations; 5. Verifying that results meet agreed requirements and quality metrics; 6. Creating and maintaining test sets (golden set) together with business and ML team, participating in quality assessment of responses; 7. Logging defects, monitoring their resolution, reporting on release quality; 8. Conducting acceptance testing before production release and participating in acceptance with business.

General experience: At least 3 years in software testing. Experience in required stack/direction: At least 1 year testing ML/LLM-based products or testing APIs and backend service integrations; understanding the specifics of non-deterministic model responses. Relevant project experience: At least 2 projects testing integration/API solutions or AI products with production deployment; experience building regression sets and test sets. Industry experience: Preferably e-commerce, retail, B2B distribution, customer service. Mandatory technologies/tools: Test design, test documentation, bug tracker; REST API testing (Postman or similar), reading JSON, logs; SQL at the level of queries for data verification; Python at a basic level for verification scripts and test data preparation; Understanding of LLM, RAG, prompts, and AI quality metrics (precision, recall, response evaluation by criteria). Preferred technologies/tools: Experience with LLM orchestration platforms (Dify, LangChain, etc.); API test automation (pytest, requests) and LLM-eval tools (RAGAS, DeepEval, promptfoo or similar); Experience labeling and evaluating model responses (LLM-as-a-judge, manual rubric evaluation); CI/CD, Git at a basic level. Task and documentation systems: Jira / Yandex Tracker, Confluence / Yandex Wiki, Git, AI initiative registry in Google Sheets.

Employment: Project-based, long-term project (full-time). Format: Remote within Russia. Location: Strictly Moscow, Saransk. Registration: GPC, IP. Salary: Up to 1400 RUB per hour gross. Important: We do not consider candidates with projects from Liman, Petrovich Tech, WB, Ozon, Lenta Tech, Yandex Market, M.Video in the last 6 months.