Job Details

Analyst-Developer role for Alice AI LLM pre-train model at Yandex. Focus on improving the model, solving applied tasks, and monitoring quality. Requires SQL, Python, and NLP metrics experience.

Our team focuses on the analytics of the Alice AI LLM pre-train model, which powers Alice, Search, and many other Yandex products. Our goal is to improve the model by solving various applied tasks, constantly monitoring quality from all angles, and ensuring nothing important is lost. To achieve this, we need a system of reliable and versatile metrics – our team is responsible for these metrics (benchmarks). What tasks await you: • Creating pre-train model quality metrics Our main task is to comprehensively evaluate the quality of the pre-train model. We often face new and interesting challenges: how to assess the model's agentic capabilities during pre-training; how to measure how well the model will perform on tasks from various Yandex services; how well our model with a high Codeforces rating will write code for users; how to correctly accept experiments on models of different sizes. • Interpreting experiment results The development team conducts many experiments and draws conclusions based on our metrics. Sometimes the results are unexpected or contradictory; in such cases, the analysts' task is to help understand what happened. • Analyzing growth points There are still complex tasks that our model does not yet handle at the proper level, but we try to select the most promising ones, analyze the problems, and, together with the development team, devise a solution plan.

We expect you to: • Have experience with metrics in NLP • Be proficient in SQL and Python • Be able to analyze data and have built effective data processing pipelines It will be a plus if you: • Have worked with LLMs • Have built agentic systems • Have created crowd-labeling with a quality control system

Yandex's voluntary medical insurance is valid from the first month of work. You can get planned or emergency care abroad – so you don't have to worry about traveling. These are not all the bonuses – the full list is here: https://yandex.ru/jobs/pages/benefits?utm_campaign=ya_nanimaet