PySpark Data Engineer – Python, ETL & Data Warehousing

Job description

Job Summary Synechron is seeking a PySpark Data Engineer with 7+ years of overall experience and at least 5+ years of commercial experience in data-driven roles . The role will design, develop, test, deploy, and support scalable data pipelines, data marts, and data warehousing solutions using Python, PySpark, SQL, and related data technologies.The position will contribute to business objectives by delivering reliable data solutions, improving data quality and accessibility, supporting analytics and reporting, and ensuring effective data processing across the full software development lifecycle. Software Requirements Required 7+ years of overall professional experience in data engineering, software development, or related technology roles. 5+ years of commercial experience in a data-driven role. Hands-on experience building data marts and ETL pipelines. Strong expertise in Python and PySpark for ETL scripting. Experience writing clean, maintainable, robust, and testable Python code. Hands-on experience with Spark, PySpark, Hadoop, MapReduce, Hive, and Pandas. Strong knowledge of SQL and Oracle query development. Experience working with SQL and NoSQL database management systems. Experience across the end-to-end software development lifecycle, including: Build and development. User acceptance testing. UAT defect resolution. Production deployment. Post-production support. Experience debugging PySpark code and investigating data processing issues. Strong understanding of data warehousing and data pipeline production practices. Ability to process structured, semi-structured, and unstructured data. Familiarity with Git, CI/CD processes, data testing, and validation. Experience with data analysis, data cleansing, data linking, imputation, and feature engineering. Familiarity with workflow orchestration and scheduling tools. Experience collaborating with multiple technical and business teams. Preferred Experience with Apache Airflow, Oozie, and Jenkins pipelines. Experience using Jupyter for data exploration, prototyping, and analysis. Knowledge of cloud-based data engineering platforms and services. Experience with data lake, lakehouse, distributed processing, and streaming concepts. Familiarity with automated data quality monitoring and pipeline observability. Experience in banking, financial services, or other regulated, data-intensive industries. Knowledge of data governance, metadata management, lineage, security, and privacy practices. Experience leading technical workstreams or coordinating delivery across multiple teams. Overall Responsibilities Design, develop, test, deploy, and support scalable ETL pipelines and data marts using Python and PySpark. Build data processing solutions for structured, semi-structured, and unstructured data. Develop clean, maintainable, robust, and reusable Python and PySpark code. Analyze business and technical requirements and translate them into data engineering solutions. Develop and optimize SQL and Oracle queries for data extraction, transformation, validation, and analysis. Integrate data from multiple sources, databases, files, and systems. Apply data cleansing, data linking, imputation, transformation, validation, and feature engineering techniques. Support data warehouse development, data modeling, data integration, and reporting requirements. Participate in build, UAT, UAT defect resolution, production deployment, and post-production support activities. Debug PySpark code, investigate pipeline failures, and resolve data quality and processing issues. Validate data outputs, reconcile results, and ensure that pipelines meet defined quality and business requirements. Collaborate with technical and non-technical stakeholders to clarify requirements, resolve dependencies, and deliver agreed outcomes. Participate in code reviews, technical discussions, testing, deployment planning, and production support activities. Identify opportunities to improve pipeline performance, automation, reliability, maintainability, and resource efficiency. Maintain technical documentation covering data flows, pipeline logic, data models, dependencies, test evidence, and operational procedures. Consider security, data privacy, cost management, and sustainability when designing and operating data solutions. Technical Skills (By Category) Programming Languages Essential Python using a current and supported version. PySpark for distributed data processing and ETL development. Strong understanding of Python functions, modules, object-oriented programming, exception handling, testing, and package management. Ability to write clean, maintainable, robust, reusable, and testable code. SQL for data extraction, transformation, validation, analysis, and query optimization. Understanding of data structures, algorithms, and software engineering principles. Preferred Shell scripting for automation and operational support. Experience developing reusable Python packages and data-processing utilities. Knowledge of programming practices for distributed and production-scale data applications. Databases/Data Management Essential Strong knowledge of relational databases and Oracle query development. Experience with SQL and NoSQL database management systems. Understanding of data warehousing, data marts, data modeling, and data integration. Knowledge of structured, semi-structured, and unstructured data processing. Experience with data cleansing, data linking, imputation, reconciliation, transformation, and validation. Understanding of data quality, data integrity, data lifecycle, and metadata requirements. Ability to analyze large datasets and identify data inconsistencies or processing issues. Preferred Experience with dimensional modeling, fact and dimension tables, and analytical data warehouse design. Knowledge of data lake and lakehouse architectures. Familiarity with data lineage, metadata management, and data governance. Experience with feature engineering and preparing data for analytics or machine learning use cases. Knowledge of database performance tuning and query optimization. Cloud Technologies Essential Understanding of cloud-based data engineering concepts and distributed data processing. Awareness of cloud storage, compute, networking, access management, monitoring, and deployment considerations. Ability to support data pipelines across development, test, UAT, and production environments. Preferred Experience developing and deploying PySpark data pipelines on cloud platforms. Familiarity with cloud-based data lakes, data warehouses, managed databases, and workflow services. Knowledge of cloud monitoring, infrastructure automation, identity management, and security controls. Understanding of cost-efficient and sustainable use of cloud data-processing resources. Frameworks and Libraries Essential Apache Spark and PySpark. Hadoop, MapReduce, and Hive. Pandas for data analysis and transformation. Python libraries for database connectivity, file handling, data validation, and automation. Experience developing ETL and data-processing frameworks. Understanding of distributed processing, partitioning, transformations, actions, and performance considerations. Preferred Apache Airflow or Oozie for workflow orchestration. Jupyter for data analysis, exploration, and prototyping. Libraries supporting data quality, testing, feature engineering, and statistical analysis. Familiarity with streaming or near-real-time data-processing frameworks. Development Tools and Methodologies Essential Experience across the end-to-end SDLC, including build, UAT, defect fixing, deployment, and post-production support. Git for source code versioning, branching, merging, and code review. Familiarity with CI/CD processes and automated build or deployment workflows. Experience with data testing, validation, reconciliation, and defect management. Knowledge of Agile or iterative software delivery practices. Ability to document data flows, transformation logic, data dependencies, test results, and operational procedures. Experience coordinating with multiple teams to resolve dependencies and deliver project outcomes. Preferred Jenkins pipeline experience. Experience with automated data quality checks and test execution. Familiarity with pipeline monitoring, logging, alerting, and incident management. Knowledge of infrastructure as code and automated environment deployment. Experience with performance monitoring and optimization of production data pipelines. Security Protocols Essential Understanding of secure data handling and data protection principles. Awareness of authentication, authorization, identity and access management, encryption, secrets management, and secure connectivity. Ability to apply appropriate access controls to data pipelines, databases, files, and processing environments. Understanding of data privacy, data integrity, auditability, and secure transfer practices. Preferred Experience implementing security controls across cloud and on-premises data environments. Knowledge of data masking, tokenization, role-based access control, and audit logging. Familiarity with vulnerability management, security testing, and compliance-related data controls. Understanding of secure configuration and monitoring practices for data platforms. Experience Requirements 7+ years of overall experience in data engineering, software development, or related technology roles. 5+ years of commercial experience in a data-driven role. Experience building data marts and ETL pipelines. Strong hands-on experience with Python and PySpark for ETL scripting. Experience with Spark, Hadoop, MapReduce, Hive, Pandas, SQL, and Oracle queries. Experience working with SQL and NoSQL database technologies. Experience across build, UAT, UAT defect resolution, production deployment, and post-production support. Experience debugging PySpark code and resolving data pipeline, data quality, and production issues. Strong understanding of data warehousing and production data pipeline practices. Experience handling structured, semi-structured, and unstructured data. Experience with CI/CD, Git, data testing, validation, workflow scheduling, and pipeline support. Experience in banking, financial services, or other regulated data-intensive industries is preferred. Candidates may also qualify through equivalent practical experience, relevant certifications, professional training, or demonstrated delivery of complex data engineering solutions. Day-to-Day Activities Design, develop, test, and maintain Python and PySpark ETL pipelines, data marts, data transformations, and data warehouse components. Collaborate with technical and non-technical stakeholders, participate in Agile meetings, clarify requirements, and resolve cross-team dependencies. Perform data analysis, Oracle query development, PySpark debugging, data validation, UAT defect fixing, and production support. Review pipeline results, monitor delivery progress, document technical outcomes, recommend improvements, and make implementation decisions within approved standards. Qualifications Degree in Computer Science, Information Technology Engineering, or an equivalent discipline; equivalent professional experience may be considered. Minimum of 7+ years of overall experience , including at least 5+ years of commercial experience in data-driven roles. Certifications in data engineering, cloud technologies, Python, Spark, Agile, or database technologies are preferred. Complete Synechron-required training related to information security, data protection, data governance, workplace conduct, and responsible technology use. Maintain continuous professional development in Python, PySpark, Spark, data warehousing, cloud data platforms, SQL, automation, security, and data engineering practices. Professional Competencies Critical thinking, data analysis, technical investigation, and structured problem-solving. Technical ownership, teamwork, dependency coordination, and delivery accountability. Clear communication with technical and non-technical stakeholders. Adaptability, continuous learning, and effective response to changing data and delivery requirements. Innovation focused on reliable, maintainable, automated, scalable, and sustainable data solutions. Effective prioritization, organization, time management, and delivery under multiple deadlines. S​ YNECHRON’S DIVERSITY & INCLUSION STATEMENT Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more. All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law . Candidate Application Notice