Data Engineer

Job description

****Position Overview:**** We are seeking a skilled Data Engineer to design and build scalable data platforms that power analytics, reporting, and business-critical insights. You will develop high-performance batch and streaming data pipelines, optimize ETL/ELT workflows, and manage large-scale analytical databases while ensuring data reliability and performance. This role requires strong expertise in SQL, distributed data processing, cloud technologies, and event-driven architectures, along with close collaboration with Product, Analytics, and Backend teams to deliver robust data solutions. At ShyftLabs, we live and breathe data. Since 2020, we’ve been helping Fortune 500 companies unlock growth with cutting-edge digital solutions that transform industries and create measurable business impact. We’re growing fast and we’re looking for passionate problem-solvers who are ready to turn big ideas into real outcomes. ### Job Responsibilities: - Design and build scalable batch and streaming data pipelines. - Develop and optimize ETL/ELT workflows using distributed data processing frameworks. - Own and optimize ClickHouse clusters for large-scale analytical workloads. - Design efficient data models for reporting and dashboarding use cases. - Build and maintain data ingestion pipelines from MongoDB, PostgreSQL, Kafka, APIs, and other data sources. - Improve performance of large SQL workloads and analytical queries. - Build reliable monitoring, health checks, and data anomaly detection systems. - Work closely with Product, Analytics, and Backend teams to deliver reliable reporting and insights. ### Basic Qualifications: - 3+ years of experience in Data Engineering. - Strong SQL skills with expertise in query optimization. - Experience with ClickHouse or other OLAP databases (BigQuery, Redshift, Snowflake, Druid, Pinot, etc.). - Strong knowledge of PostgreSQL. - Experience building ETL/ELT pipelines. - Proficiency in Java or Python. - Experience with Apache Kafka and event-driven architectures. - Strong understanding of data modeling and partitioning strategies. - Experience working on cloud platforms (AWS/GCP/Azure). - Experience with Docker and Kubernetes. - Knowledge of monitoring and observability tools. ### Preferred Qualifications: - Experience with Looker or BI platforms. - Experience with Apache Spark or Dataproc. - Experience with Airflow or workflow orchestration tools. - Understanding of advertising technology (DSP, RTB, Attribution, Campaign Reporting). - Experience with large-scale analytical systems processing billions of records.