Senior Technical Lead
HCL Technologies Limited
Dubai, AE · Dubai · United Arab Emirates
Posted Nov 16, 2026
Apply by Dec 20, 2026
- Other
Job description
Job Summary A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data. Core Responsibilities Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses. Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data. Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling. Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC ). Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational). CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing. Technical Skills Required Category Skills & Technologies Core Languages Python (Advanced), SQL (Advanced) Big Data Frameworks Apache Spark / PySpark , Hadoop, Hive Cloud Platforms AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc) Storage & Formats Delta Lake , Parquet, Avro, HDFS, Snowflake, Amazon S3 Orchestration Apache Airflow , Prefect, or Luigi Key Responsibilities as above Skill Requirements as above Other Requirements