Senior Databricks and Snowflake Platform Administrator

Job description

Analytical Wizards is part of the Definitive Healthcare family. We balance innovation with an open, friendly culture and the backing of a long-established parent company, known for its ethical reputation. We guide customers from what’s now to what’s next by unlocking the value of their data and applications to solve their challenges, achieving outcomes that benefit both business and society. Our people are our biggest asset, they drive our innovation advantage and we strive to offer a flexible and collaborative workplace where they can thrive. We offer industry-leading benefits packages to promote a creative and inclusive culture. If driving real change gives you a sense of pride and you are passionate about powering social good, we’d love to hear from you. ## Position Summary The Senior Databricks & Snowflake Platform Administrator is responsible for the administration, security, governance, automation, and operational excellence of the enterprise data platform. This role owns Databricks workspaces, Unity Catalog, Snowflake environments, access management, metadata governance, infrastructure-as-code implementation, platform optimization, and cost management. The administrator serves as the primary custodian of data platform governance, ensuring secure self-service access for engineering and data delivery teams while maintaining centralized control of catalogs, schemas, databases, permissions, and platform standards. ## Key Responsibilities ### Databricks Platform Administration - Administer Databricks workspaces across development, test, and production environments. - Manage clusters, SQL Warehouses, serverless resources, jobs, workflows, and workspace configurations. - Configure and maintain Unity Catalog, external locations, storage credentials, and data access controls. - Monitor platform health, performance, utilization, and costs. - Implement security controls, audit logging, and compliance requirements. - Partner with Engineering teams to establish platform standards and best practices. ### Snowflake Administration - Administer Snowflake accounts, warehouses, databases, schemas, roles, resource monitors, and security policies. - Manage workload performance, query optimization, and warehouse sizing. - Implement Snowflake governance, access controls, and secure data-sharing processes. - Manage integrations between Snowflake, Databricks, cloud storage, and downstream applications. - Perform capacity planning and cost optimization activities. ### Infrastructure as Code (Terraform) **In cooperation with DevOps team, own and maintain Terraform-based platform administration.** Responsibilities include: - Create and maintain Terraform modules for: - Unity Catalog catalogs - Schemas/databases - External locations - Storage credentials - Snowflake databases and schemas - Data platform infrastructure - Enforce platform ownership boundaries: - Platform team owns catalogs, schemas, databases, storage credentials, permissions, and governance controls. - Engineering teams own only: - Tables - Views - Materialized Views - Data pipelines and code - Establish GitOps-based deployment pipelines for all administrative changes. - Ensure all platform configuration changes are version-controlled and auditable. ### Identity & Access Management (Terraform Driven) **Own enterprise access governance for Databricks and Snowflake.** Responsibilities include: - Implement role-based access control (RBAC) using Terraform. - Manage user provisioning, group assignments, and entitlement policies. - Maintain Azure AD / Entra ID group integration. - Provision: - Unity Catalog grants - Catalog privileges - Schema privileges - Snowflake roles and grants - Workspace permissions - Eliminate manual access assignment wherever possible. - Conduct periodic access reviews and remediation. - Maintain least-privilege access standards. ### Unity Catalog Ownership & Governance **Serve as the primary owner of Unity Catalog governance.** Responsibilities include: - Own catalog lifecycle management. - Define and enforce naming standards. - Manage catalog and schema ownership models. - Maintain metadata governance standards. - Implement data classification and tagging frameworks. - Enforce governance policies across environments. - Manage: - External locations - Storage credentials - Catalog permissions - Cross-workspace data access - Drive cleanup and rationalization of unused assets. - Establish processes for catalog onboarding and retirement. ### Hive Metastore Cleanup & Rationalization **Lead migration and cleanup initiatives to modernize Databricks governance.** Responsibilities include: - Inventory legacy Hive Metastore assets. - Identify: - Orphaned tables - Stale schemas - Unused databases - Duplicate datasets - Abandoned workloads - Lead migration of assets into Unity Catalog. - Implement archival and retention policies. - Remove obsolete datasets and metadata. - Reduce cloud storage costs through lifecycle management and cleanup activities. - Improve metadata quality and catalog discoverability. - Publish governance and cleanup metrics. ### Data Governance & Compliance - Establish and enforce enterprise data governance standards. - Support audit and compliance requirements. - Implement data lineage and metadata management practices. - Maintain documentation for governance policies and platform standards. - Partner with Security and Compliance teams on platform reviews. ### Platform Reliability & Operations - Manage incident response and root-cause analysis. - Develop operational runbooks and support procedures. - Maintain SLAs for platform availability and support. - Implement automated monitoring and alerting. - Plan platform upgrades and migrations. ## Required Qualifications - 5+ years of Databricks administration experience. - 5+ years of Snowflake administration experience. - Strong expertise with Unity Catalog governance and security. - Strong Terraform and Infrastructure-as-Code experience. - Experience with Azure, AWS, or multi-cloud environments. - Deep understanding of: - RBAC - IAM - Entra ID / Azure AD - SSO - SCIM provisioning - Experience automating platform operations via Python, Terraform, and CI/CD pipelines. - Strong SQL and data platform architecture knowledge. - Experience with data governance, cataloging, and metadata management. ## Preferred Qualifications - Databricks Certified Data Engineer or Databricks Platform certifications. - Snowflake SnowPro Administrator certification. - Experience with GitOps deployment models. - Experience with FinOps and cloud cost optimization. - Experience managing data platforms supporting AI/ML workloads. - Experience leading enterprise-scale governance initiatives. ## Success Metrics - 100% Terraform-managed platform configurations. - 100% Terraform-managed user access and role assignments. - Reduction in manual access requests and permission exceptions. - Reduction in orphaned Unity Catalog and Hive Metastore assets. - Measurable cloud storage cost reduction through cleanup initiatives. - Improved governance compliance and audit readiness. - Improved platform uptime and operational efficiency. - Successful migration of legacy Hive Metastore assets into Unity Catalog.