Forward Deployed Data Engineer IV, Databricks

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Forward Deployed Data Engineer IV, Databricks based in United States.** This is a senior data engineering role focused on solving complex operational challenges for utility clients through modern data platforms and production-grade systems. You’ll work directly in client environments to turn fragmented and complex data into governed, reliable, and usable foundations for analytics, applications, and AI. The role spans the full lifecycle, from technical discovery and architecture through development, deployment, and handover. You’ll use Databricks, Spark, Python, SQL, and AWS to build scalable data platforms and products that deliver measurable outcomes. You’ll collaborate closely with client technical teams and executives as well as data scientists, engineers, and consultants. The environment values ownership, rapid delivery, technical judgment, clear communication, and practical solutions that remain sustainable after an engagement ends. ### Accountabilities: - Embed directly with utility clients to design and build production data platforms covering ingestion, transformation, orchestration, quality, governance, and serving using Databricks, Spark, Python, SQL, and AWS. - Own the technical architecture of client engagements from discovery through deployment, clearly communicating and defending architectural decisions with client architects, security teams, and platform owners. - Lead technical discovery independently, challenge assumptions when appropriate, identify underlying business needs, and recommend solutions that address the actual problem rather than simply the requested implementation. - Deliver useful working systems early in engagements, typically within the first few weeks, and progressively harden them for production based on real-world usage and operational requirements. - Lead migrations from legacy data warehouses and on-premises environments to modern cloud platforms, including identifying and addressing undocumented dependencies. - Develop robust data models using dimensional, lakehouse, medallion, graph, or semantic approaches according to the requirements of each domain and use case. - Integrate utility systems such as asset and work management platforms, historians and SCADA systems, customer information systems, advanced metering infrastructure, GIS, and ERP platforms through APIs, change data capture, file-based integrations, and other available mechanisms. - Build observable and operable pipelines with lineage, data quality controls, alerting, monitoring, and runbooks that client engineering teams can effectively maintain. - Develop operator-facing applications such as data quality dashboards, reconciliation interfaces, review and correction tools, and self-service data applications, using Databricks Apps or comparable technologies where appropriate. - Manage engagement scope, timelines, expectations, and delivery risks against defined statements of work, escalating issues early and maintaining alignment with stakeholders. - Contribute reusable ingestion frameworks, engineering patterns, and reference architectures that can be applied across client engagements while sharing field insights with internal product and engineering teams. - Apply appropriate data governance, security, privacy, lineage, and retention controls within the regulatory and operational context of the utility industry. ### Requirements: - Hold a Bachelor’s degree in Computer Science, Information Technology, or a related field, with a Master’s degree in a relevant STEM discipline considered an advantage. - Bring 5+ years of experience in data engineering, data platforms, or analytics engineering, including ownership of at least one production data platform. - Demonstrate expert proficiency in Python, SQL, Databricks, and Spark, including an understanding of Spark runtime behavior and the ability to diagnose performance issues without simply increasing compute resources. - Have experience building and operating cloud-based data pipelines in AWS, with working knowledge of an additional cloud ecosystem considered a plus. - Demonstrate strong knowledge of core data engineering challenges, including incremental processing, full rebuilds, idempotency, late-arriving data, deduplication, backfill strategies, and schema evolution. - Bring experience designing both batch and streaming pipelines, with the judgment to determine which architecture best addresses the underlying business and technical requirements. - Have practical knowledge of data governance, security, privacy, lineage, and data quality practices, together with proficiency in Git, Docker, CI/CD tooling, and at least one modern orchestration framework. - Demonstrate experience using AI coding assistants and agentic development tools to rapidly prototype, test, and iterate on production-oriented software; this is a particularly important capability for the role. - Be capable of owning the full system lifecycle, from technical discovery and architecture through development, deployment, and post-go-live support. - Have experience working directly with enterprise clients and navigating competing priorities among technical contributors, business stakeholders, and senior executives. - Demonstrate strong project delivery skills, including managing technical scope, timelines, risks, and measurable outcomes in fast-moving engagements. - Communicate clearly through written documentation and technical whiteboarding, and remain effective when working with ambiguity, incomplete data, and changing objectives. - Experience in the energy, utility, or another asset-intensive industry is valuable, particularly familiarity with the operational systems, data environments, and constraints common to these sectors. - Experience with graph databases and knowledge graph modeling, Kafka or equivalent streaming platforms, change data capture tooling, ML/AI feature pipelines, Databricks Apps, React, Streamlit, Dash, REST or GraphQL APIs, or Databricks performance optimization is advantageous. - A Databricks certification is considered a plus. - Must be authorized to work for any employer in the United States; employment visa sponsorship or transfer of sponsorship is not available for this role. ### Benefits: - Budgeted salary of $175,000–$200,000 USD plus an annual bonus, with actual compensation adjusted based on experience. - Medical, dental, and vision insurance options. - Company-paid life insurance. - Company-paid short-term and long-term disability insurance. - Medical and dependent-care flexible spending accounts. - Paid parental leave. - Flexible Time Off (FTO), subject to manager approval and business coverage requirements. - 401(k) plan with a 3% employer match. - Remote-based work within the United States. - Significant client travel, generally 30–50% depending on the engagement, including potential extended on-site periods during discovery, deployment, and go-live. - Opportunities to develop technical expertise while working on meaningful data and technology challenges within the utility and energy sector.