Senior Technical Lead
HCL Technologies Limited
Posted Nov 20, 2026
Apply by Dec 22, 2026
- Other
Job description
Job Summary Role: Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics. Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers. Carry out data modelling in Databricks. Expose the Delta tables to the visualisation layer. Ensure that the data pipelines are optimised. Skillsets : Expert in Apache Spark Structured Streaming and Databricks Auto Loader. Fluent in PySpark and Databricks SQL. Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control. Key Responsibilities Role: Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics. Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers. Carry out data modelling in Databricks. Expose the Delta tables to the visualisation layer. Ensure that the data pipelines are optimised. Skillsets : Expert in Apache Spark Structured Streaming and Databricks Auto Loader. Fluent in PySpark and Databricks SQL. Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control. Skill Requirements Role: Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics. Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers. Carry out data modelling in Databricks. Expose the Delta tables to the visualisation layer. Ensure that the data pipelines are optimised. Skillsets : Expert in Apache Spark Structured Streaming and Databricks Auto Loader. Fluent in PySpark and Databricks SQL. Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control. Other Requirements Role: Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics. Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers. Carry out data modelling in Databricks. Expose the Delta tables to the visualisation layer. Ensure that the data pipelines are optimised. Skillsets : Expert in Apache Spark Structured Streaming and Databricks Auto Loader. Fluent in PySpark and Databricks SQL. Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control.