Director of Data Engineering & Platform

Job description

Director of Data Engineering & Platform Cooley is seeking a Director of Data Engineering & Platform to join the Data team within the Innovation department. Position summary: As a leading technology law firm, Cooley is determined to become a leader in the digital practice of law. The Director of Data Engineering & Platform is a leadership role responsible for building, leading, and operating the Data Engineering & Platform function across Data Engineering and Platform & Operations. This role joins the Northstar Lakehouse program at a critical build phase, inheriting a live platform that is actively moving data into the Silver layer, with core Data Vault 2.0 structures being established across the firm's primary legal sector entities. The Director will develop high-performing Data Engineering and Platform Operations teams that deliver the Northstar Lakehouse from Silver layer foundations to gold layer products serving the firm’s AI and analytics ambitions. The platform is built on Databricks on AWS with a medallion architecture, Data Vault 2.0 methodology for non-financial data domains, and a two-track gold layer serving both legal intelligence and financial reporting consumption patterns. This is a hands-on leadership role that will build a greenfield data platform moving quickly, building correctly, and delivering at the pace a legal AI program demands. Specific duties and responsibilities include, but are not limited to, the following: Position responsibilities: Data Engineering & Platform Leadership: Lead and develop the Innovation Data Engineering & Platform function across Data Engineering and Platform & Operations. Build a high-performing, accountable team through clear expectations, consistent feedback Provide direct leadership to the team and as the function matures and team members develop, make deliberate decisions regarding introducing a management layer beneath the Director Direct contract resources including transitional Enterprise Solution Architects and Contractor program resources, setting clear work agendas, managing delivery accountability, and making recommendations on resource extension or transition based on platform needs and team capability development Partner with the Director of Data on function strategy, hiring sequencing, budget, and the long-term capability roadmap for the Data Engineering & Platform function as the Northstar Lakehouse transitions from build phase to operational maturity Develop the Senior Data Engineering Manager as the technical engineering leader this function needs over time. Provide direct, frequent feedback on delivery quality, team management, and technical decision-making Own workforce planning for the function, including permanent hire sequencing, contractor-to-permanent conversion decisions, and the right-sizing of the team as the platform moves from construction to steady-state operations Serve as direct supervisor and mentor to direct reports Provide day-to-day supervision of direct reports, ensure compliance with assigned work hours and monitor for compliance with all firm and department policies. Manage staffing coverage, review and process time logs/time off requests Support business professional development and continued educational opportunities In collaboration with immediate supervisor and central HR, participate in hiring, performance appraisals, counseling, termination and other employee lifecycle events Northstar Lakehouse Build & Technical Delivery: Own the technical delivery of the Northstar Lakehouse build across the Bronze, Silver, and Gold layers of the Databricks medallion architecture. Drive reliable, high-quality data pipeline delivery across Data Vault 2.0 structures for non-financial legal sector domains and 3NF relational structures for the 3E financial and billing data domain Establish and maintain the two-track gold layer architecture: a Data Vault Business Vault consumption layer serving legal intelligence, client and matter analytics, competitive intelligence, and AI-powered legal products, and a 3NF relational reporting layer serving financial reporting, billing analytics, and operational dashboards from the 3E legal billing and matter management system Ensure the Data Engineering team implements Data Vault 2.0 pipelines in alignment with the methodology standards defined by the Principal Data Architect in the Platform Governance & Quality function. Create the conditions for the Principal Data Architect to provide effective methodology oversight, code review, and standards enforcement without friction Drive the Bronze layer ingestion quality baseline, ensuring source-to-platform data completeness, reconciliation back to source systems, and structural validation are established across all active data sources before Silver layer certification is approved Own the delivery cadence of the Data Engineering function within the squad-based EPIC delivery model. Ensure Agile ceremonies are effective, sprint commitments are realistic and met, and the engineering team is unblocked and productive across active Data Domain Squads Partner with the Director of Data Platform Governance & Quality on the certification workflow for Bronze and Silver layer data assets, ensuring data quality gates, governance controls, and access policies are embedded in the engineering build process rather than applied as post-delivery audits Platform Architecture & Operations: Own the Databricks platform architecture decisions for the Northstar Lakehouse, including Unity Catalog metastore configuration, compute tier design, job cluster and SQL warehouse architecture, Delta Lake table property standards, workspace organization, and platform security configuration Direct the Platform Architect in maintaining and evolving the Databricks platform as capabilities change, new Databricks features become available, and the platform's consumption patterns grow. Ensure the platform architecture stays current, performant, and aligned with the firm's AI and analytics ambitions Own the CI/CD framework for data pipeline deployment, ensuring reliable and automated promotion of pipeline code through development, UAT, and production environments. Hold the DataOps Engineer accountable for CI/CD framework operation and evolution as the pipeline estate scales Oversee Platform & Operations reliability across the live Databricks environment, including compute health, pipeline execution monitoring, incident management, and escalation protocols across a he platform that is live and serving active data consumers. Operational reliability is a must from Day 1 Manage the integration of external tools and data sources with the Databricks platform, including Qlik Replicate for CDC ingestion, gateway and VM infrastructure coordination, and any additional tooling integrations required as the platform scales Technology Infrastructure Relationship & Cross-Functional Collaboration: Own the relationship with the firm's Technology function as the senior accountability point for all infrastructure dependencies that sit outside the Databricks platform. This includes network connectivity, gateway configuration, VM management, data center and cloud infrastructure coordination, and security policy alignment between the platform and the broader technical environment Work closely with the Technology function on infrastructure dependencies for the Data Engineering & Platform team. Proactively surface infrastructure requirements early, build credible relationships with Technology counterparts, and resolve cross-functional blockers at the appropriate level without escalating unnecessarily Partner with the Director of Data Platform Governance & Quality on cross-functional platform standards including data certification workflows, Unity Catalog governance configuration, access control policies, and the technical governance standards the Platform Governance & Quality function defines and the Engineering team implements Collaborate with the Director of Data Products and the Intelligence & Client Services function on gold layer data product delivery, ensuring the engineering team understands the analytical and AI consumption requirements that drive gold layer design decisions and delivers the data structures those products depend on Engage with the Data Council through the Data Platform Governance & Quality function, providing engineering perspective on platform capability, delivery timelines, and technical feasibility of governance-driven data requirements surfaced by Data Domain Squads and working groups All other duties as assigned or required Required: After orientation at Cooley LLP, exhibit proficiency in the Microsoft Office suite, iManage, and other firm applications Ability to work extended and/or weekend hours, as required Ability to travel, as required 10+ years of experience in data engineering, data platform architecture, or enterprise solution architecture with demonstrated leadership of a data engineering function in a cloud-native environment with 5+ years of management experience in relevant roles Proven experience building a greenfield cloud data platform from the ground up, including hands-on involvement in medallion architecture design, Data Vault 2.0 implementation, pipeline framework establishment, and gold layer consumption pattern development Deep Databricks expertise including Unity Catalog configuration and governance, Delta Lake table design and optimization, Databricks SQL and job cluster architecture, compute tier strategy, workspace organization, and Databricks platform administration at an enterprise scale Strong Data Vault 2.0 literacy, sufficient to assess whether Raw Vault hub, link, and satellite implementations are methodologically correct Experience designing and delivering a two-track or multi-layer gold layer architecture serving both analytical BI consumption and AI-powered product consumption from the same Silver layer foundations Demonstrated experience directing ESA, contract, and transitional resources alongside a permanent engineering team, setting clear work agendas, managing delivery accountability, and making informed decisions about resource retention and transition Proven track record of effective leadership Experience owning cross-functional infrastructure relationships, specifically coordinating between a data platform engineering team and an IT infrastructure or enterprise technology function on network, gateway, VM, and cloud connectivity dependencies Proven Agile delivery leadership in a squad-based data platform program, including sprint planning, EPIC delivery management, and definition of done discipline across multiple parallel data domain workstreams Comfort using GitHub for version control, CI/CD pipeline oversight, and engineering standards governance in a collaborative data platform environment Bachelor’s degree Preferred: Legal sector, professional services, or financial services experience with understanding of complex operational data environments including matter management, client relationship data, timekeeper and billing data, or comparable professional services data complexity Experience with 3NF relational modeling for operational or ERP systems, specifically in the context of building a financial or operational reporting gold layer alongside a Data Vault analytical layer Experience with dbt for transformation layer implementation including model development, testing frameworks, documentation standards, and the governance of dbt as a platform-wide transformation standard Familiarity with Qlik Replicate, AWS DMS, or comparable CDC ingestion tooling for real-time or near-real-time data replication from operational source systems into a cloud data platform Experience with Informatica MDM, DQE, or comparable enterprise master data and data quality platforms and how they integrate with a Databricks lakehouse environment Familiarity with infrastructure-as-code tooling such as Terraform, AWS CloudFormation, or comparable for Databricks workspace and resource management AI-assisted development fluency including Claude Code, GitHub Copilot, or comparable tools used to accelerate engineering delivery, code review, and documentation workflows across the team Experience designing or operating a data platform that serves as the foundation for RAG pipelines, LLM fine-tuning datasets, vector search indexes, or comparable AI infrastructure Competencies: Strong platform builder instincts with a genuine understanding of what it takes to move a greenfield data platform from foundations to gold layer products that business stakeholders can trust and consume Excellent people leadership skills with a demonstrated ability to develop engineering talent, hold a team accountable to high standards, and make honest promotion and performance decisions without avoidance Deep technical credibility in Databricks and cloud data platform architecture, sufficient to make real-time architectural decisions, assess engineering team work quality, and direct experienced ESA resources effectively Strong collaborative instincts with the ability to build productive working relationships across governance, data products, intelligence, and IT infrastructure functions without creating friction or competing for authority Excellent judgment about when to be hands-on versus when to delegate, understanding that a greenfield program in build phase sometimes requires the Director to roll up their sleeves alongside the team Strong delivery orientation with a track record of meeting commitments, running effective Agile ceremonies, and driving a data engineering team to consistent, high-quality output under program pressure Excellent written and verbal communication skills with the ability to represent the Data Engineering & Platform function credibly to executive stakeholders, the Technology function, and cross-functional platform partners Growth-oriented talent developer with genuine investment in the Senior Data Engineering Manager's career trajectory and the broader team's technical capability development Strong attention to detail Strong judgment High level of professionalism at all times Demonstrated ability to lead through influence and develop talent [if applicable] Unwavering ability to handle and maintain confidentiality regarding firm information, projects, and client data Cooley offers a competitive compensation and excellent benefits package and is committed to fair and equitable employment practices. EOE. The expected annual pay range for this position with a full-time schedule is $260,000 - $320,000. Please note that final offer amount will be dependent on geographic location, applicable experience and skillset of the candidate. We offer a full range of elective benefits including medical, health savings account (with applicable medical plan), dental, vision, health and/or dependent care flexible spending accounts, pre-tax commuter benefits, life insurance, AD&D, long-term care coverage, backup care for children and/or adults and other parental support benefits. In addition to elective benefit options, benefited employees receive firm-paid life insurance, AD&D, LTD, short term medical benefits as well as 25 days of Paid Time Off (“PTO”) and 10 paid holidays each year. We provide generous parental leave and fertility benefits. New employees will attend a detailed benefit orientation to learn more about our many benefits and resources.