Senior Developer in LLM Inference Optimization Team at Avito

Avito's LLM Inference Optimization team seeks a Senior Developer to optimize LLM inference on GPU, enhance hardware efficiency, and assist product teams in deploying models.

Responsibilities: - Conduct load testing of models per product team requests and optimize for specific usage scenarios. - Develop performance evaluation methodology for LLMs on our hardware: benchmarks, tools, and monitoring. - Regularly measure performance, track new model releases, and maintain a results table. - Test new GPUs and oversee their configuration with infrastructure and inference platform teams. - Scale optimizations across the model fleet: speculative decoding, multi-LoRA, distributed inference, transition to new inference backends. - Drive changes to production following the cycle: profile, optimize, test, deploy with confirmed metrics. - Run LLM models using the inference platform.

Requirements: - Experience optimizing LLM models based on load profiles. - Worked with LLM inference frameworks: vLLM, SGLang, TensorRT-LLM, and others; launched and configured them, adjusting parameters. - Understanding of LLM inference internals and optimization methods: prefill, decode, KV cache, batching, speculative decoding, quantization. - Knowledge of GPU principles: memory, bandwidth, utilization, NVLink, multi-GPU configurations. - Profiled GPUs and used monitoring tools. - Conducted load testing and benchmarking of LLM models: latency, TTFT, TPOT, throughput, etc., with correct result comparison and conclusions. - Experience with Docker, Kubernetes, Prometheus, Grafana metrics and dashboards. - Willingness to deploy changes to production, maintain them, and explain results to product teams. Nice to have: - Wrote or optimized CUDA kernels; understand SIMT and NVIDIA GPU architecture. - Implemented speculative decoding, multi-LoRA serving, or distributed inference. - Worked with routing and load balancing considering KV cache: llm-d, NVIDIA Dynamo, Mooncake, or similar. - Know C++ or Go. - Follow open-source articles and projects on inference acceleration and quickly test ideas in practice.

Conditions: - Office in Moscow. - Extended health care: voluntary health insurance, life insurance, cancer insurance. - Psychological support service, plus financial, legal, and healthy lifestyle consultations for you and your family. - Corporate sports: group workouts, wellbeing app. - Special offers from partners and discounts on Avito services: accommodation booking, promotion services, mortgage insurance. - Meal compensation. - Engineering freedom to implement large-scale tasks. - Product-driven approach to infrastructure based on metrics. - Strong engineering community for support and experience sharing. - Effective growth tools: competency matrix, internal training, mentoring, and experience exchange.