Job Details

Senior Data Engineer for a long-term government project in Abu Dhabi, UAE. Relocation support provided. Focus on building a Data Lakehouse and pipelines for AI projects. Requires Python, SQL, Spark, Kafka, Airflow, Databricks, SAP integration experience. English B2+.

Building a modern Data Lakehouse and pipelines for unstructured data within a large-scale AI project for Supply Chain and engineering workflows. Working in a strictly regulated and secure environment. Tech Stack: Languages: Python, SQL. Data Stack: Spark, Kafka, Airflow, Databricks / Delta Lake. Integrations: SAP S/4HANA, SAP Ariba (SAP BTP — plus), PLM systems. AI Ops: Vector DBs (Weaviate / Milvus), pipelines for RAG. Infrastructure: Docker, Kubernetes, CI/CD, dbt / Dagster / Prefect. Responsibilities: — Design and deliver Data Lakehouse architecture as a single source of truth for analytics and AI. — Build ingestion pipelines from corporate systems (SAP, PLM) with near real-time data availability. — Integrate external market feeds to enrich procurement and operations data. — Process unstructured data (PDF tenders, drawings, technical documentation) into formats ready for vector search (RAG). — Optimize vector databases for fast information retrieval. — Implement Governance: audit, RBAC, Data Lineage, automated data quality checks (BOM, cost).

Must-have: — 5+ years of Data Engineering experience (2+ years with pipelines for ML/GenAI/RAG). — Solid experience with Spark, Kafka, and Airflow. — Proven experience integrating with SAP systems (S/4HANA, Ariba). — Practical skills with Databricks and PostgreSQL. — Willingness to undergo a strict security background screening. Nice to have: — Knowledge of Supply Chain / Manufacturing domain (procurement lifecycle, BOM). — Experience working in closed/restricted network environments. — Understanding of Digital Thread and data traceability concepts.

Relocation: Full support for moving. Minimum first 6 months onsite, then hybrid or remote is possible upon agreement. Format: Full-time (long-term project from Government). Start: ASAP.