Job Details

Company: Megafon. Python Developer in Moscow. Responsibilities include Python development, ETL process design and support, working with Apache Kafka, PostgreSQL, and ClickHouse.

• Python development • Design, development, optimization, and support of ETL processes for data loading from various sources • Managing real-time data processing using Apache Kafka (developing producers/consumers, managing topics, ensuring reliability and fault tolerance) • Development and optimization of data structures, schemas, and queries in PostgreSQL for operational and transactional tasks • Building, administering, and fine-tuning ClickHouse clusters to ensure high performance of analytical queries on large data volumes • Development and support of solutions based on ClickHouse (or a PostgreSQL + ClickHouse combination) • Data integration between PostgreSQL, Kafka, ClickHouse, and other infrastructure components (APIs, queues, S3) • Monitoring the performance, fault tolerance, and security of data pipelines and databases • Automation of data processing and deployment processes (CI/CD) • Collaboration with analysts, developers, and product owners to understand data needs and implement them • Troubleshooting data pipelines and databases

• 3+ years of experience in Python development • Experience in ClickHouse development and administration (understanding table engine specifics, query optimization, clustering, replication, sharding) • Deep understanding and practical experience with Apache Kafka, configuring and ensuring data stream reliability • Experience with PostgreSQL (writing complex SQL queries, performance optimization, understanding transaction mechanisms, replication) • Understanding of ETL process design principles • Advanced SQL knowledge • Understanding of DevOps principles (Docker, Kubernetes, Linux, CI/CD) in the context of data infrastructure