Senior Data Engineer

Job description

Business Area: IT Seniority Level: Mid-Senior level Job Description: At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world’s largest enterprises. Cloudera Data Engineers power our Analytics and AI/ML initiatives by building scalable, high-performance data pipelines. In this role, you will lead our transition from traditional manual coding to AI-orchestrated development (Vibe Coding), architecting next-gen data pipelines and GenAI applications at unprecedented speed. You will focus on modernizing development workflows and building GenAI-powered self-service tools that empower the business to resolve data needs independently. By designing robust, AI-first data management processes on Cloudera’s native platform, you will ensure data integrity while creating a blueprint for both internal efficiency and external customer success. As a Senior Data Engineer you will: Collaborate with Data Architects, Operational Architects, and Data Analysts to understand the data and operational requirements across different business units. Partner with data owners to ensure seamless, reliable data ingestion for both traditional analytics and GenAI-powered applications. Master "Vibe Coding" and AI-orchestrated development to accelerate the delivery of new data pipelines and GenAI applications, reducing the end-to-end development lifecycle from days to hours. Develop and implement data transformations to enrich and provision data, following established specifications and standards while utilizing AI-first workflows. Design and implement robust system architectures for real-time, near real-time, and batch processing data flows to meet the operational demands of complex business systems. Design and deploy GenAI-powered "Self-Service" tools, including automated documentation generators and natural language interfaces, to empower business users and reduce routine engineering requests. Implement monitoring and CI/CD automation processes to track data quality and ensure the reliability of AI-supported data services. Standardize AI-first engineering workflows across the team to ensure high-quality, auto-validated, and well-documented code delivery. We are excited if you have (Required Experience): 5+ years of experience as a Data Engineer. Proven experience with AI-first approaches and "Vibe Coding," with a demonstrated ability to deliver production-ready data pipelines using AI orchestration rather than purely manual coding. Deep proficiency with AI-assisted coding tools, including Cursor, GitHub Copilot, or Gemini, to modernize and accelerate engineering workflows. Solid skills in System Design for diverse data architectures, including expert-level knowledge of batch processing and real-time/streaming processing. Proficient in coding with Python (primary) and SQL, with experience in ETL and data processing. Hands-on experience with Distributed Systems and Big Data technologies, including Spark and the Hadoop ecosystem (Hive, Impala, Kafka). Proven proficiency in Data Modeling using industry best practices (e.g., Kimball, Inmon) to ensure data integrity. Ability to monitor critical data pipelines for quality and resolve any issues effectively. Education: Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field or equivalent experience. Strong communication skills, both written and verbal. You may also have: Experience supporting GenAI applications from the Data Engineering side, including managing vector databases, RAG (Retrieval-Augmented Generation) pipelines, or LLM data orchestration. Experience with Apache Airflow or Apache NiFi. Expertise in optimizing data storage using HDFS/Parquet/Avro, Kudu, or HBase. What you can expect from us: Generous PTO Policy Support work life balance with Unplugged Days Flexible WFH Policy Mental & Physical Wellness programs Phone and Internet Reimbursement program Access to Continued Career Development Comprehensive Benefits and Competitive Packages Paid Volunteer Time Employee Resource Groups EEO/VEVRAA #LI-HYBRID #LI-EO1