AI & Data Platform Architect

Job description

Principal AI & Data Architect – Enterprise Data Platform (AI, Analytics & GenAI) Role Summary: We are seeking a Senior AI & Data Platform Architect with deep expertise in Azure Databricks, Snowflake, cloud-scale data engineering, Lakehouse architecture, and enterprise analytics platforms. In this role, you will define, design, and govern enterprise-scale data processing, analytics, and AI-ready platform solutions across Azure Databricks and Snowflake. You will lead architectural decisions across multiple initiatives, ensuring alignment with business objectives, cloud strategy, security, performance, scalability, reliability, governance, and cost optimization standards. The ideal candidate will combine strong hands-on architecture experience, platform engineering mindset, stakeholder leadership, and the ability to guide delivery teams on modern data platform implementation patterns. This role will be accountable for designing end-to-end Databricks and Snowflake solutions, establishing architecture standards, reviewing solution designs, mentoring senior engineers, and improving overall data platform maturity across the organization. Roles & Responsibilities • Act as the architecture lead and subject matter expert for Azure Databricks, Snowflake, and enterprise data platform initiatives across projects and programs. • Analyze business, data, functional, and non-functional requirements and translate them into scalable end-to-end architecture designs. • Design modern data lakehouse and data warehouse architectures using Azure Databricks, Snowflake, Azure Data Lake Storage Gen2, Delta Lake, and Azure Data Factory. • Define and govern medallion architecture standards across raw, curated, and serving layers, including bronze, silver, and gold data patterns. • Architect scalable ingestion, transformation, orchestration, and consumption patterns across batch, incremental, near-real-time, and streaming workloads. • Define Databricks cluster architecture, job design, workflow orchestration, notebook standards, and workload isolation strategies. • Design Snowflake warehouse strategies, data modelling patterns, workload management, query optimization, cost controls, and secure data sharing approaches. • Establish reusable reference architectures, solution patterns, guardrails, and engineering standards for Databricks, Snowflake, and AI-ready data products. • Drive performance optimization, scalability, resiliency, and cost efficiency through cluster tuning, SQL optimization, resource management, and platform monitoring. • Define data security and governance practices including RBAC, ABAC, secrets management, encryption, row-level and column-level access, masking, lineage, and compliance controls. • Review solution designs, notebooks, SQL assets, pipelines, infrastructure templates, and deployment approaches for architectural compliance. • Collaborate with enterprise architects, cybersecurity, cloud infrastructure, data governance, platform operations, analytics, and business stakeholders on key design decisions. • Guide CI/CD, DevOps, and release management strategies for Databricks and Snowflake deployments using Git-based workflows and automated promotion patterns. • Provide architectural guidance during production incidents, root cause analysis, platform optimization, capacity planning, and operational maturity initiatives. • Mentor technical leads, senior data engineers, and platform engineers through architecture reviews, design walkthroughs, and best-practice enablement. Professional & Technical Skills – Must Have • Strong hands-on and architectural experience with Azure Databricks, including cluster configuration, cluster policies, job scheduling, workflow orchestration, notebook design, and workload optimization. • Advanced expertise in Apache Spark architecture, PySpark, Spark SQL, Spark performance tuning, partitioning, caching, shuffle optimization, and scalable data processing patterns. • Deep understanding of Delta Lake, Delta tables, schema evolution, time travel, optimization, vacuuming, and lakehouse reliability patterns. • Strong proficiency in SQL for data transformations, data modeling, analytics workloads, and performance tuning. • Strong experience with Snowflake architecture, including virtual warehouses, resource monitors, workload isolation, Snowpipe, Streams, Tasks, Snowpark, secure data sharing, cloning, and Time Travel. • Experience designing scalable data models across lakehouse, data warehouse, dimensional, Data Vault, semantic, and consumption-oriented architectures. • Deep understanding of data engineering, data warehousing, ELT/ETL, metadata management, orchestration, and data quality frameworks. • Experience implementing secure, scalable, governed, and AI-ready data platforms using Azure Databricks, Snowflake, Azure Data Lake Storage Gen2, and modern cloud-native services. • Strong knowledge of Azure cloud security, identity, access control, networking, private endpoints, secrets management, and governance controls. • Experience defining CI/CD and deployment strategies for Databricks notebooks, jobs, workflows, libraries, infrastructure-as-code, and Snowflake database objects. • Ability to evaluate architectural trade-offs across performance, cost, scalability, maintainability, security, reliability, and delivery speed. • Strong communication skills with the ability to influence enterprise architects, security teams, platform teams, engineering teams, and senior stakeholders. Preferred Qualifications • Experience designing enterprise-scale data platforms that support analytics, reporting, machine learning, Generative AI, and agentic AI use cases. • Exposure to Unity Catalog, catalog design, data ownership models, metadata management, lineage, and governance operating models. • Experience with Snowflake Cortex, Snowpark, external functions, vector search, semantic search, or AI-enabled analytics capabilities is preferred. • Experience integrating Databricks and Snowflake with BI, ML, MLOps, LLMOps, data catalog, observability, and enterprise monitoring tools. • Knowledge of streaming and event-driven architectures using Kafka, Event Hubs, structured streaming, Snowpipe Streaming, or equivalent technologies. • Experience with platform modernization, migration from legacy data warehouses, cloud data lake implementation, or large-scale data product enablement. • Familiarity with cost optimization practices across Databricks compute, Snowflake warehouses, storage tiers, orchestration, and consumption workloads. • Relevant certifications such as Databricks Data Engineer Professional, Databricks Solutions Architect, SnowPro Core, SnowPro Advanced Architect, or Microsoft Azure Solutions Architect are desirable. Additional Information • The candidate should have 9–12 years of experience in data engineering, analytics, big data platforms, or cloud data architecture, with clear architecture ownership across enterprise initiatives. • This position is based at Bengaluru, Chennai, or Hyderabad office locations. • Minimum 15 years of full-time education or equivalent qualification is required. • The role is responsible for driving architecture governance, technical standards, platform scalability, platform stability, cost optimization, and engineering maturity across multiple teams. • The architect will be accountable for enabling robust, secure, high-performance, and governed Databricks and Snowflake solutions aligned to enterprise data and AI strategy. Key Success Measures • Enterprise architecture standards for Databricks and Snowflake are defined, communicated, and adopted across delivery teams. • Data pipelines and analytics workloads are scalable, secure, performant, cost-efficient, and aligned with governance requirements. • Reusable platform patterns, medallion architecture standards, CI/CD practices, and deployment guardrails improve delivery consistency. • Production incidents, performance bottlenecks, and platform risks are reduced through proactive design reviews and operational governance. • Senior engineers and technical leads are mentored to improve architecture quality, platform maturity, and engineering excellence.