Senior Databricks Data Engineer (Data Migration Project)

Job description

Founded in 1994 and headquartered in Switzerland, ERNI is a leading Software Development company with over 800 employees worldwide. Specializing in IT and software engineering, we drive innovation in process and technology. Our first service center in Asia Pacific, located in Metro Manila (Mandaluyong), supports clients across Europe, APAC, the Philippines, and the USA. As we continue to grow, we're looking for passionate and motivated individuals to join our team. Why ERNI is the Perfect Place for You: 🏡 • International Exposure: Work with global clients on cutting-edge projects. • Inclusive Culture: Thrive in a collaborative and diverse work environment. • Career Development: Enjoy continuous learning and professional growth opportunities. 🤩Perks and Benefits: • Career Stability: Enjoy a stable career path with ample project opportunities. • Immediate Coverage: Private HMO and insurance benefits from day one. • Jubilee Celebration: A 5-year milestone includes a complimentary trip to any European ERNI sites . • Comprehensive Benefits: Government-mandated benefits including 13th-month pay. • Skill Enhancement: Access free training and certifications. • Baby Basket: To welcome your newborn to the ERNI family. • Fruit Basket: Boost of vitamins during hospitalization. • Office Perks: Enjoy free snacks and coffee. 🔐Growth and Opportunities: • Free Training: Advance your skills through technical and non-technical training. • Challenging Projects: Engage in complex software projects across MedTech, Industry, Finance, and Transportation. • Supportive Environment: Benefit from a team dedicated to guiding and supporting your success. • Recognition and Advancement: Receive acknowledgment for your efforts and opportunities for promotion. • Open Communication: Experience transparency and value your input in our culture. ⏱Flexibility: • Hybrid Work Setup: Balance remote and in-person work for better work-life integration. 🎉Events: • Connect and Celebrate: Participate in a variety of events including leisure, summer, family, social, and year-end gatherings. 👋What are our wishes: Minimum 8 years of professional data engineering experience, including hands-on design and delivery of enterprise-scale data pipelines, data platforms, ETL/ELT processes, and analytics-ready datasets. Senior-level hands-on experience with Azure and Azure Databricks is required. Strong experience developing data engineering solutions using Databricks, PySpark, Spark SQL, Delta Lake, and SQL. Strong understanding of cloud-native data engineering patterns, including scalable ingestion, transformation, orchestration, monitoring, performance tuning, and production support. Experience building and maintaining data pipelines that process large volumes of structured, semi structured, and unstructured data from multiple source systems. Strong experience with data modeling, data marts, curated data layers, and business-facing datasets for reporting, analytics, and operational consumption. Strong ability to clarify requirements directly with stakeholders and translate business rules, data definitions, and process needs into technical implementation details. Experience with source-to-target mapping, data profiling, data validation, reconciliation, and data quality management. Experience with job orchestration and scheduling tools such as Azure Data Factory, Databricks Workflows, Airflow, or similar technologies. Strong understanding of data governance, access control, metadata, lineage, security, and compliance considerations in a cloud data environment. Experience with performance tuning and optimization of Spark jobs, Databricks clusters, SQL queries, and Delta tables. Strong troubleshooting, problem-solving, and root-cause analysis skills in production data environments. Strong communication skills, with the ability to explain technical concepts, risks, dependencies, and delivery options to business and technology stakeholders. Ability to work independently with minimal supervision while collaborating effectively in cross-functional delivery teams. Preferred / Nice-to-Have Qualifications Experience supporting near-real-time, frequent refresh, or low-latency data/application refresh patterns. Experience with Spark Structured Streaming, Auto Loader, Kafka, Event Hubs, change data capture, incremental ingestion, or streaming tables is preferred but not mandatory. Experience designing lakehouse-style data layers such as raw, cleansed, conformed, curated, and consumption-ready layers. Experience with CI/CD, DevOps, automated testing, version control, deployment pipelines, and environment management for data engineering solutions. Experience with Azure services such as Azure Data Lake Storage, Azure Data Factory, Azure Key Vault, Azure Monitor, Azure Purview / Microsoft Purview, Azure Synapse, or related services. Experience working in regulated, data-sensitive, or enterprise environments with strong governance, audit, and security requirements. Experience supporting business intelligence, analytics, machine learning, or operational reporting use cases. Technical Skill Expectations Required: Azure, Azure Databricks, Delta Lake, PySpark, Spark SQL, SQL, ETL/ELT development, data pipeline design, data modeling, data quality controls, production support, performance tuning, and stakeholder requirements clarification. Strongly preferred: Azure Data Lake Storage, Azure Data Factory, Databricks Workflows, Git, CI/CD, DevOps practices, orchestration frameworks, metadata management, monitoring, and cost/performance optimization. Nice to have : Structured Streaming, Auto Loader, Kafka, Event Hubs, CDC, incremental processing, near-real-time refresh patterns, and low-latency data pipelines. 💼How can you contribute to the team? The IT Senior Data Engineer will design, build, optimize, and support scalable data engineering solutions on a modern cloud-native data platform. The role will focus on developing reliable, secure, and high performing data pipelines, curated data layers, and analytics-ready data assets that support business operations, reporting, advanced analytics, and new strategic initiatives. This role requires a senior-level data engineer who can work independently across technical and business teams, clarify requirements directly with stakeholders, translate business needs into technical delivery plans, and ensure that data products are designed for quality, performance, reusability, and long-term maintainability. Key Responsibilities Design, develop, and optimize automated data pipelines, ETL/ELT workflows, data models, and curated data layers using Azure, Azure Databricks, Delta Lake, PySpark, SQL, and related cloud-native data engineering tools. Build scalable ingestion, transformation, validation, and publishing processes to support enterprise data consumption across reporting, analytics, application, and operational use cases. Partner directly with business stakeholders, product owners, data consumers, analytics teams, and subject matter experts to clarify business requirements, data definitions, transformation rules, data quality expectations, refresh requirements, and acceptance criteria. Translate business requirements into technical specifications, source-to-target mappings, data models, pipeline designs, delivery estimates, and implementation plans. Facilitate requirement-discovery discussions, playback sessions, technical walkthroughs, and solution reviews with both technical and non-technical stakeholders. Design and implement reliable data marts, analytical datasets, curated data products, and customized data extracts aligned with business priorities and consumption patterns. Develop reusable data engineering frameworks, patterns, and components that improve delivery speed, operational reliability, monitoring, and maintainability. Ensure data pipelines and data products are designed with appropriate controls for data quality, lineage, auditability, security, privacy, access management, and compliance. Implement validation checks, reconciliation processes, exception handling, logging, and monitoring to ensure data accuracy, completeness, and timeliness. Monitor pipeline health, performance, cost, and reliability; troubleshoot production incidents; perform root cause analysis; and implement preventive and corrective actions. Optimize Databricks jobs, Spark workloads, SQL queries, Delta tables, partitioning strategies, and orchestration patterns for performance, scalability, and cost efficiency. Work closely with data architects, platform engineers, product owners, data analysts, and data scientists to ensure that data infrastructure and data products are reliable, discoverable, well-governed, and fit for business use. Support batch and scheduled data processing requirements, with experience in near-real-time or frequent application/data refresh patterns considered a strong advantage. Contribute to data architecture decisions involving lakehouse design, data layering, ingestion patterns, transformation standards, orchestration, metadata management, and operational support. Other details: Set up: Hybrid Schedule: Dayshift - 9AM - 6PM Office location: Mandaluyong / BGC (client's office)