Sr Data Engineer, Data Analytics & Intelligence, NA

Job description

About Vantage Data Centers Vantage Data Centers powers, cools, protects and connects the technology of the world’s well-known hyperscalers, cloud providers and large enterprises. Developing and operating across North America, EMEA and Asia Pacific, Vantage has evolved data center design in innovative ways to deliver dramatic gains in reliability, efficiency and sustainability in flexible environments that can scale as quickly as the market demands. Operational Excellence Data Team Within Operational Excellence, the Data Analytics & Intelligence function enables Operations to move from reactive reporting to proactive, insight-driven execution. The team is responsible for building trusted data foundations, governed KPI frameworks, operational intelligence products, and AI-ready data assets that support performance visibility, decision-making, predictive insights, and scalable operational excellence across North America. This work directly supports Operational Excellence's mission of embedding delivery rigor, process discipline, and data intelligence into how Operations plans, executes, predicts, and continuously improves. Position Overview This position will be based on-site at our office in Denver, CO. in alignmen t with our flexible work policy. (3 days on site required, 2 days flexible). Vantage Data Centers is seeking a Sr Data Engineer to help build, operate, and scale the governed data foundation for Operations, North America. This role is designed for an engineer who can independently deliver production-ready pipelines, curated datasets, semantic-model inputs, and AI-ready data products that support reporting, Executive reporting insight preparation, and the emerging AI Insight Solution. As part of the Data Engineering & Business Intelligence team, you will be responsible for delivering reliable data products that support analytics, AI data agents, operational intelligence, reporting, and an emerging AI-enabled platform. You will work closely with IT Global, solution build teams, business SMEs, and data governance partners to ensure data products are secure, reusable, explainable, and aligned with enterprise AI / Fabric direction. Success in this position requires comfort with ambiguity, strong execution discipline, and accountability for building trusted data assets that can be reused across analytics, operational intelligence, and AI-enabled use cases. Essential Job Functions Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform. Develop and operate batch and incremental data pipelines leveraging Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store. Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs that support governed operational insights and AI-enabled consumption. Independently implement SQL- and Spark-based transformations to produce curated datasets that support enterprise reporting, analytics, AI-enabled insight preparation, and downstream consumption. Take ownership of assigned data pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support. Work with Azure Synapse, Microsoft Fabric / Lakehouse patterns where applicable, and related Azure analytics services to support analytical workloads and data consumption patterns. Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions. Support source visibility, confidence context, and Data Reliability & Trust Indicator integration where applicable so downstream analytics and AI outputs can be understood and trusted. Contribute to ontology, taxonomy, semantic model, and data dictionary alignment needed to connect operational context, KPIs, incidents, work orders, and other enterprise data domains. Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into practical, working data solutions. Apply established data governance, security, access-control, data classification, and engineering standards to ensure compliant, maintainable, and scalable solutions. Identify, document, and route data-quality issues to accountable owners, helping improve source correction rather than masking defects downstream. Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives as an active contributor. Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate them clearly in a fast-paced environment. Duties Develop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform. Build and modify Azure Data Factory pipelines for batch and incremental data ingestion. Implement Spark-based transformations that write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards. Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs to support analytics, operational intelligence, and AI-enabled consumption patterns. Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions. Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with clear root-cause analysis and practical remediation steps. Perform performance tuning of Spark jobs and SQL workloads, including partitioning, filtering, incremental logic, query optimization, and resource-aware design within established architectural patterns. Validate data outputs with business partners, operations SMEs, and data stewards, and address defects or discrepancies through documented correction paths. Support observability, logging, and auditability practices for data pipelines and AI-consumable datasets where applicable. Commit code using Git, follow branching standards, participate in pull request reviews, and support CI/CD ways of working using GitHub, Azure DevOps, or similar tools. Update documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made. Execute assigned backlog items within sprint timelines and raise risks, dependencies, or blockers early. Additional duties as assigned by management. Job Requirements Education & Experience Bachelor’s degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience. Minimum of 5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role. Proficiency in Python for building and maintaining data pipelines, automation, data processing workflows, and PySpark-based transformations. Proficiency in SQL for querying, transformation, analytical data processing, model validation, and data quality checks. Solid understanding of ETL/ELT pipelines, data transformation patterns, data integration concepts, incremental processing, and production support practices. Experience analyzing enterprise data sources to identify data relationships, transformations, business rules, grain, ownership, and quality constraints. Experience building solutions on the Microsoft Azure platform with exposure to Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, Microsoft Fabric / Lakehouse patterns, and related analytics services. Working knowledge of data modeling fundamentals, including fact and dimension tables, semantic models, reusable data products, and analytics-ready structures. Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, data-quality validation, and operational runbooks. Experience working with source control and CI/CD workflows using tools such as GitHub or Azure DevOps. Strong communication and interpersonal skills with the ability to collaborate across IT Global, business SMEs, data governance partners, platform teams, and operations stakeholders in a fast-paced environment. Experience working in Agile development environments and using collaboration or project tracking tools such as Jira or similar tools. Travel required is expected to be up to 10% but may increase over time as the business evolves. Desired Qualifications Experience working with distributed data processing frameworks, including Apache Spark. Experience preparing governed data products for AI-enabled use cases, including Microsoft Fabric Lakehouse, semantic models, Fabric/Data Agent patterns, ontology or taxonomy alignment, and explainable AI outputs. Familiarity with data observability, metadata management, lineage, data contracts, reliability indicators, and operational best practices in production environments. Familiarity with additional Azure services such as Azure Functions or Logic Apps in support of data workflows. Experience supporting data platform enhancement, refactoring, modernization, or reusable architecture initiatives. Experience working with structured and unstructured operational sources, such as enterprise applications, operational workflows, documents, dashboards, and knowledge assets. Experience working in a scaling or fast-paced organization where priorities evolve quickly and practical delivery discipline is required. Physical Demands and Special Requirements The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. While performing the duties of this job, the employee is occasionally required to stand; walk; sit; use hands to handle, or feel objects; reach with hands and arms; climb stairs; balance; stoop or kneel; talk and hear. The employee must occasionally lift and/or move up to 25 pounds. Additional Details Salary Range: $ 130k -155k (this range is based on Colorado market data and may vary in other locations) This position is eligible for company benefits including but not limited to medical, dental, and vision coverage, life and AD&D, short and long-term disability coverage, paid time off, employee assistance, participation in a 401k program that includes company match, and many other additional voluntary benefits. Compensation for the role will depend on a number of factors, including your qualifications, skills, competencies, and experience and may fall outside of the range shown. #LI-Hybrid We operate with No Ego and No Arrogance. We work to build each other up and support one another, appreciating each other’s strengths and respecting each other’s weaknesses. We find joy in our work and each other, actively seeking opportunities to inject fun into what we do. Our hard and efficient work is rewarded with an above market total compensation package. We offer a comprehensive suite of health and welfare, retirement, and paid leave benefits exceeding local expectations. Throughout the year, the advantage of being part of the Vantage team is evident with an array of benefits, recognition, training and development, and the knowledge that your contribution adds value to the company and our community. Don't meet all the requirements? Please still apply if you think you are the right person for the position. We are always keen to speak to people who connect with our mission and values. Vantage Data Centers is an Equal Opportunity Employer Vantage Data Centers does not accept unsolicited resumes from search firm agencies. Fees will not be paid in the event a candidate submitted by a recruiter without an agreement in place is hired; such resumes will be deemed the sole property of Vantage Data Centers. We’ll be accepting applications for at least one week from the date this role is posted. If you're interested, we encourage you to apply soon—we’re excited to find the right person and will keep the role open until we do!