Lead Data Engineer

Job description

# # Job Title: Lead Data Engineer **Experience:** 5+ Years **Location:** Remote (Anywhere in India) **Employment Type:** Full-time --- ## About the Role We are seeking a **Lead Data Engineer** with 5+ years of hands-on experience designing and building **scalable, reliable data platforms and pipelines**. The ideal candidate will have strong technical expertise across the modern data stack, experience owning data engineering initiatives end-to-end, and the ability to translate business and analytical requirements into robust data solutions. You will be responsible for **architecting, designing, and delivering data engineering solutions**, while also mentoring engineers and driving best practices across data architecture, data quality, governance, and performance. This is a hands-on technical leadership role where you will work closely with Product, Engineering, Analytics, and Business stakeholders to build a scalable data foundation that supports reporting, analytics, and self-service BI. --- ## Key Responsibilities - Design, develop, and manage **scalable ETL/ELT pipelines** for structured and unstructured data using AWS services such as **AWS Glue, Redshift, S3, Athena, Lambda, and Kinesis/Kafka**. - Architect and implement **data lake and data warehouse solutions**, following best practices for data modeling, governance, scalability, and performance. - Own the **technical design and end-to-end delivery** of data engineering initiatives. - Develop robust data models and pipelines that support **analytics, reporting, and business intelligence** use cases. - Drive **data pipeline automation, orchestration, monitoring, and observability** using tools such as **Apache Airflow and AWS Step Functions**. - Enable **self-service analytics** by designing and curating data models and semantic layers that empower product owners and business users to build their own visualizations using Generative BI tools such as **ThoughtSpot, QuickSight Q, or similar platforms**. - Optimize data pipelines, queries, and warehouse performance to ensure scalability, reliability, and cost efficiency. - Establish and maintain standards for **data quality, integrity, security, governance, and compliance** across the data lifecycle. - Collaborate with stakeholders across **Product, Engineering, Analytics, and Business** to translate requirements into scalable and maintainable data solutions. - Identify opportunities to improve existing data architecture, processes, and infrastructure through automation and engineering best practices. - Mentor junior and mid-level engineers and contribute to building a **high-performing data engineering team**. - Stay current with emerging technologies and best practices in **data engineering, cloud platforms, data architecture, and analytics**. --- ## Required Skills & Qualifications - **5+ years of professional experience in Data Engineering**, with demonstrated ownership of complex data engineering projects. - Strong proficiency in **Python and SQL**. - Hands-on experience with ETL/ELT frameworks and orchestration tools such as **AWS Glue, Apache Airflow, and AWS Step Functions**. - Strong experience with the **AWS data stack**, including **S3, Glue, Redshift, Athena, Lambda, and Kinesis or Kafka**. - Proven experience designing and implementing **data lakes, data warehouses, and scalable data pipelines**. - Strong understanding of **data modeling, dimensional modeling, data warehousing, and database design**. - Experience with **query optimization, performance tuning, and cost optimization** across data platforms. - Experience with **Spark/PySpark and/or EMR** for large-scale data processing. - Experience enabling **self-service analytics** through well-designed data models and semantic layers. - Hands-on experience with BI/analytics platforms such as **ThoughtSpot, Amazon QuickSight, or similar tools**. - Strong understanding of **data quality, governance, security, and data lifecycle management**. - Excellent problem-solving, communication, stakeholder management, and **technical leadership skills**. --- ## Preferred Qualifications - Experience **leading a technical team** or mentoring data engineers. - Experience owning data engineering projects from **architecture and design through implementation and production**. - Experience working in a **SaaS or product-based environment**. - Familiarity with modern data architecture patterns, including **data lakes, lakehouses, streaming pipelines, and event-driven architectures**. - Experience with **Kafka/Kinesis** and real-time or near-real-time data pipelines. - AWS certifications in **Data Engineering, Solutions Architecture, or related areas**. - Contributions to open-source projects or demonstrated involvement in the broader data engineering community.