Senior Databricks Platform Engineer

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Databricks Platform Engineer based in United States.** This role offers the opportunity to lead the design, administration, and optimization of an enterprise-scale Databricks Lakehouse Platform. You will help build modern data engineering, governance, analytics, and AI/ML capabilities supporting a large-scale federal modernization initiative. The position combines hands-on platform engineering with performance optimization, automation, security, and data governance. You will work across cloud infrastructure, distributed data processing, observability, and enterprise data architecture. Your expertise will directly contribute to scalable, reliable, secure, and cost-efficient data platforms. The environment offers significant technical ownership, collaboration with architecture and security teams, and opportunities to influence platform standards and engineering practice ### Accountabilities: - Design, configure, administer, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, scalable data architectures, clusters, SQL warehouses, serverless capabilities, and workload management. - Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows to support reliable and scalable data processing. - Establish and govern Unity Catalog structures, including catalogs, schemas, tables, roles, groups, RBAC, and access-control policies aligned with security, IAM, and compliance requirements. - Automate platform provisioning, configuration, testing, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs while supporting consistent infrastructure-as-code practices. - Monitor platform health and performance through logging, alerting, observability, capacity planning, and KPI dashboards, while troubleshooting platform, pipeline, and data-processing issues. - Perform Spark and SQL performance tuning, scalability testing, workload optimization, and root-cause analysis to improve reliability, responsiveness, and overall platform efficiency. - Implement cost-management strategies such as autoscaling, auto-termination, right-sizing, workload isolation, and capacity forecasting while contributing to DBU consumption analysis, financial reporting, and total-cost-of-ownership assessments. - Support platform upgrades, patching, version management, release processes, and adoption of new Databricks capabilities that can improve performance, cost efficiency, security, or operational maturity. - Integrate Databricks with enterprise data sources, governance platforms, business intelligence tools, and AI/ML environments to create an integrated data ecosystem. - Implement and maintain security, encryption, networking, auditing, compliance, high-availability, and disaster-recovery controls across the platform. - Support security audits, authorization-to-operate activities, incident response, operational reporting, and root-cause investigations while ensuring appropriate governance and documentation. - Establish reusable engineering patterns, platform standards, reference architectures, runbooks, and technical documentation that promote consistency and operational excellence. - Collaborate closely with architecture, security, governance, and data engineering teams while providing technical guidance and mentorship to data engineers and platform users. - Evaluate emerging Databricks features and technologies, recommend appropriate adoption strategies, and continuously identify opportunities to improve platform performance, scalability, cost, and operability. ### Requirements - A **Master’s degree in information systems or a related field with 12+ years of general experience and 10+ years of specialized experience** is preferred. Equivalent experience may be considered in lieu of a degree, including 18+ years with a high school diploma, 16+ years with an associate degree, 14+ years with a bachelor’s degree, or 11+ years with a doctorate. - Strong hands-on experience implementing, administering, and optimizing the **Databricks Lakehouse Platform**, including Unity Catalog, cluster management, SQL warehouses, and job or workflow orchestration, is required. - Demonstrated expertise in **Databricks performance engineering, scalability testing, Spark optimization, SQL tuning, and workload management** is essential for this role. - Proficiency in **Python, Bash, and SQL** is required for automation, testing, scripting, data engineering, and platform administration activities. - Experience with data ingestion, ETL/ELT, lakehouse and warehouse architectures, Apache Spark, distributed data processing, and modern data engineering practices is expected. - Experience operating Databricks in **AWS-based or hybrid cloud environments** is required, along with an understanding of cloud infrastructure, resource governance, and enterprise data platforms. - Hands-on experience with **CI/CD and DevOps tooling**, including Jenkins, GitLab CI, GitHub Actions, or Azure DevOps, as well as version control and infrastructure-as-code automation, is important. - Experience with observability and performance monitoring tools such as **Grafana, Prometheus, Datadog, CloudWatch, or Elastic** is strongly valued. - Familiarity with Docker, Kubernetes, microservices patterns, telemetry, logging standards, traceability, and root-cause analysis is beneficial for operating modern cloud data platforms. - Strong knowledge of cloud security, IAM, access controls, networking, encryption, auditing, and resource governance within Databricks and Unity Catalog is required. - Familiarity with **federal security, compliance, and audit processes**, including ATO and FedRAMP environments, is highly valuable. - Strong analytical and troubleshooting skills are required, including the ability to analyze large performance datasets, identify trends and bottlenecks, and recommend optimization opportunities. - Excellent communication, collaboration, and mentoring skills are important for working effectively with technical teams, security stakeholders, architects, governance groups, and platform users. - Preferred certifications include **Databricks Certified Data Engineer Associate or Professional, Databricks Certified Associate Platform Administrator, Databricks Certified Machine Learning Professional, Databricks Certified Associate Developer for Apache Spark, AWS Certified Data Analytics, AWS Certified Solutions Architect, AWS Certified Data Engineer, Docker Certified Associate, DevSecOps, ITIL, or AWS SysOps** certifications. ### Benefits - **Salary range of $169,604–$229,464**, with final compensation determined by experience, geographic location, and applicable contractual requirements; actual compensation may fall outside this range. - **Fully remote work** with the flexibility to work from any location in the United States. - **Full-time schedule of 40 hours per week** with no travel required. - Medical plan options, including plans with **Health Savings Accounts**, along with dental and vision coverage. - A **401(k) retirement plan** offering pre-tax and post-tax contribution options and an employer match. - Flexible work arrangements and a range of paid leave programs designed to support work-life balance. - Paid vacation, sick and personal leave, holidays, parental leave, military leave, bereavement leave, and jury duty leave, with new employees typically receiving 15 paid leave days per calendar year plus 10 paid holidays. - Up to **160 hours of paid family leave** during a rolling 12-month period for eligible employees. - Additional financial protection through short- and long-term disability, life insurance, accidental death and dismemberment, personal accident, critical illness, and business travel and accident coverage. - The opportunity to work on a **large-scale cloud data modernization program** supporting data engineering, governance, analytics, and AI/ML capabilities. - Significant exposure to **Databricks, AWS, Spark, Delta Lake, Unity Catalog, Terraform, observability, automation, and enterprise data architecture**. - Opportunities to provide technical leadership, establish engineering standards, mentor data professionals, and influence the evolution of a modern data platform.