Sr Data Specialist - R01564133

Job description

**Lead Data Engineer** ### Primary Skills - Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front ### Specialization - AWS Data EngineerIng Basic: Senior Data Engineer ### Job requirements --- **Job Description – AWS Senior Data Engineer** **Company:** Brillio Technologies **Role:** Senior Data Engineer **Experience:** 5–10 Years **Location:** India (Hybrid/Remote) **Employment Type:** Full-time --- **Role Overview** We are looking for a **Senior Data Engineer with strong Databricks and Big Data expertise** to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with **Spark, PySpark, AWS ecosystem, and modern data architectures**, along with exposure to APIs and real-time data processing. --- **Key Responsibilities** - Design and build **scalable data pipelines** using Spark / PySpark on Databricks - Develop and optimize **ETL/ELT workflows** for large-scale data processing - Work with **AWS EMR, S3, and Hadoop ecosystem** for distributed data processing - Build and maintain **data lake and data warehouse solutions** - Develop and integrate **APIs for data ingestion and consumption** - Implement data processing workflows using **Airflow / Autosys** - Optimize data performance using **partitioning, caching, and query tuning** - Handle structured and semi-structured data from multiple sources - Ensure **data quality, governance, and reliability** of pipelines - Collaborate with cross-functional teams (Analytics, Product, Engineering) --- **Technical Skills (Must-Have)** - Strong expertise in: - **Apache Spark / PySpark** - **Databricks (mandatory)** - **SQL (PostgreSQL or similar)** - Hands-on experience with: - **AWS EMR, S3** - **Hadoop, Hive ecosystem** - Programming skills: - **Python (must-have)** - **Scala (good exposure)** - Strong knowledge of: - **UNIX / Shell scripting** - **ETL pipelines and data engineering fundamentals** --- **Additional Skills** - Experience with: - **Elasticsearch (data storage & retrieval)** - **Workflow orchestration tools (Airflow, Autosys)** - Exposure to: - **API development & integration** - Version control tools (**Git / SVN**) - Basic knowledge of **HTML** (for web/data tasks) --- **Experience Required** - Minimum **5+ years of Data Engineering experience** - Strong hands-on experience in **Databricks + Big Data stack** - Proven experience working on **large-scale distributed systems** --- **Education** - Bachelor’s or Master’s degree in **Computer Science, Engineering, or related field** --- **Why Join Brillio** - Work on **cloud-native, large-scale data platforms** - Exposure to **cutting-edge Databricks & AWS ecosystems** - Collaborative and high-performance engineering culture