Airflow Platform Engineer

Job description

Airflow Platform Engineer1 Experience - 4 - 6 years Key Responsibilities DAG Development & Developer Enablement • Design, develop, and optimize Airflow DAGs for enterprise use cases • Demonstrate strong experience with operators such as PythonOperator, SQL operators, API/HTTP operators, KubernetesPodOperator etc. • Apply advanced DAG patterns including dynamic DAGs, branching, sensors, retries, and error handling • Troubleshoot DAG failures, performance issues, and integration challenges • Enable orchestration of modern data workflows including DBT pipelines • Provide reusable templates, best practices, and onboarding support to drive self-service adoption Platform Operations & Kubernetes Support • Support Airflow platform deployed on AKS, covering scheduler, webserver/API, workers, and metadata services • Troubleshoot issues related to task execution, scheduling delays, and resource bottlenecks • Diagnose Kubernetes-level issues involving pods, networking, storage, and RBAC • Work with Azure services such as Key Vault, Storage, and networking • Support CI/CD pipelines for DAG and container-based deployments • Improve platform stability, monitoring, and alerting Required Skills • Strong experience with Apache Airflow (3.x or later) in production environments • Proven experience in DAG development using operators and orchestration patterns • Working knowledge of Kubernetes (AKS preferred) to support Airflow runtime • Expert-level Python knowledge for building, debugging, and maintaining production- grade workflows and supporting libraries • Expertise in writing unit, integration, and DAG validation tests for Airflow workflows, including mocking operators, validating dependencies, and ensuring reliability in CI/CD pipelines • Experience with DBT and modern data pipelines • Experience with CI/CD tools (Git, Jenkins, Docker) • Familiarity with Azure or other cloud platforms Preferred Skills • Experience with Astronomer / Astro Private Cloud • Exposure to data platforms such as Snowflake, and APIs • Experience with monitoring tools such as Splunk or Prometheus • Knowledge of secrets management using Azure Key Vault Summary This role is critical to ensuring reliable platform operations and scalable adoption of the Airflow-based orchestration platform. The ideal candidate combines strong Airflow expertise, Kubernetes knowledge, and hands-on development experience to support both platform stability and developer productivity.