Senior Data Engineer (3rd Party Contractor)

Job description

Job Title: Senior Data Engineer Reports to: Practice Head: Data, AI and Automation ROLE OVERVIEW As a Senior Data Engineer, you will play a pivotal role in designing, building, and optimizing data systems and pipelines for DVT's clients. You will be responsible for ensuring the seamless flow of data between systems, enabling efficient analytics, and supporting business intelligence efforts. Your work will involve collaborating with cross-functional teams to understand data needs, designing scalable solutions, and implementing data management best practices. Additionally, you will mentor junior engineers, contribute to the strategic vision for data architecture, and stay updated with emerging data engineering technologies. TECHNICAL KNOWLEDGE Strong knowledge and extensive experience in: · Programming Languages: Python, Java, Scala, and SQL. · Data Engineering Tools: Apache Spark, Apache Beam, Kafka, and Airflow. · Big Data Technologies: Hadoop, Hive, and Presto. · ETL/ELT Tools: AWS Glue, Azure Data Factory, and Talend. · Data Warehousing: Redshift, Snowflake, BigQuery, and Synapse Analytics. · Cloud Platforms: AWS, Azure, GCP, and Databricks. · Version Control & CI/CD: Git, Jenkins, Docker, Kubernetes, and Terraform. Strong knowledge and understanding of: · Data modeling, schema design, and query optimization. · Building and maintaining real-time data streaming pipelines. · Implementing data governance, quality, and security frameworks. · Data serialization formats such as Avro, Parquet, and ORC BEHAVIOURAL COMEPTENCIES Self-motivated and proactive with strong problem-solving abilities. Detail-oriented and highly organized with a focus on delivering high-quality solutions. Comfortable working in fast-paced, dynamic environments. Adaptable to change and ambiguity with a flexible mindset. Strong collaboration and teamwork skills, with excellent written and verbal communication. Demonstrates leadership potential and the ability to mentor others. Passion for learning and keeping up with data engineering advancements. Business acumen with the ability to align technical solutions to business goals RESPONSIBILITIES Design, build, and maintain scalable data pipelines and systems. Develop and optimize complex ETL/ELT workflows for high-volume datasets. Troubleshoot and resolve data pipeline and system issues efficiently. Participate in architecture and strategic planning sessions for data initiatives. Communicate complex technical concepts to non-technical stakeholders clearly. Document data engineering processes and contribute to the development of standards. Mentor and guide junior and intermediate data engineers in their technical growth. Implement best practices for data management, quality, security, and governance. Collaborate with data scientists, analysts, and stakeholders to ensure data solutions meet business requirements. MINIMUM EXPERIENCE NEEDED TO BE SUCCESSFUL IN THIS ROLE 8+ years of experience as a Data Engineer or in a similar role. Certifications in cloud platforms (e.g., AWS, Azure, GCP) or data engineering tools are a plus. Strong track record of implementing robust, efficient, and scalable data solutions . Proven experience with large-scale data systems and distributed computing. Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field. A Master’s degree is preferred.