Azure Databricks Data Engineer

Job description

We are looking for an experienced Azure Databricks Data Engineer to help build and operate scalable, production-grade data platforms on Azure Databricks and Microsoft Azure. The ideal candidate combines strong Spark engineering expertise with a production engineering mindset, treating data pipelines as reliable, enterprise-grade products rather than ad-hoc solutions. This role requires hands-on experience delivering secure, governed, high-performance batch and streaming data platforms at scale. Tasks Design, build, and maintain scalable batch and streaming pipelines using bronze/silver/gold architectures Develop reusable transformation frameworks and production-grade Spark workloads Implement and optimize Delta Lake solutions, including: ACID transactions, Partitioning strategies, Schema evolution, OPTIMIZE/ZORDER, Incremental and CDC processing patterns Configure and manage Unity Catalog permissions, governance, lineage, and auditability Perform Spark performance tuning and observability monitoring Integrate Databricks with Azure services including ADLS, ADF/Synapse, Key Vault, and Microsoft Entra ID Implement CI/CD pipelines and deployment automation using Azure DevOps or GitHub Actions Collaborate with architects, analysts, data scientists, and engineers to align technical implementation with business needs Continuously improve platform performance, reliability, cost efficiency, and standardization Requirements Strong hands-on expertise with Azure Databricks engineering Advanced Apache Spark experience using: PySpark or Scala Spark SQL Deep understanding of Delta Lake architecture and optimization techniques Experience building scalable enterprise data platforms and real-time pipelines Strong governance and security experience in multi-team environments Experience with service principals, managed identities, lineage, and auditability Strong troubleshooting and Spark performance optimization skills Azure integration experience across storage, orchestration, and security services Solid CI/CD and Infrastructure-as-Code exposure Comfortable working independently in consulting-style delivery environments