Big Data Platform Engineer

Experience Requirements ● 8+ years of overall IT experience ● 5+ years of experience in Cloud Infrastructure Engineering ● 4+ years of experience working with Kubernetes platforms ● 3+ years of experience supporting enterprise-scale Data Platforms ● 3+ years of experience in DevOps/SRE practices ● 2+ years of experience working with Azure Data Services Mandatory Skills Candidates must possess strong hands-on experience in the following areas: ● Apache NiFi ● Azure Event Hub ● Azure Functions ● Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered. Requirements1. Azure Kubernetes Service – AKS Strong Expertise ● AKS cluster deployment and administration ● Cluster upgrades and patch management ● Node pool management ● Ingress Controllers, including NGINX and Application Gateway ● RBAC implementation ● AKS security hardening ● Autoscaling using HPA, VPA, and Cluster Autoscaler ● Persistent storage management ● Troubleshooting cluster issues ● Network policies Experience ● Production support of AKS environments ● Multi-node and multi-environment AKS setups ● High availability and disaster recovery implementation 2. Kubernetes and Azure Networking Kubernetes Networking ● Kubernetes networking concepts ● Services, Ingress, and Load Balancers ● DNS resolution ● Pod-to-Pod communication ● Pod-to-Service communication ● Network Policies ● Service Mesh concepts, preferably Istio ● CNI ArchitectureAzure Networking ● Virtual Networks ● Subnets ● Private Endpoints ● Peering ● Network Security Groups ● User Defined Routes ● Azure Firewall ● VPN connectivity 3. Docker and Containerization ● Docker image creation and optimization ● Container lifecycle management ● Registry management ● Security scanning ● Multi-stage builds ● Container troubleshooting ● Azure Container Registry ● Image versioning and governance 4. Azure DevOps and CI/CD CI/CD Expertise ● Azure DevOps Pipelines ● YAML pipelines ● Build and Release pipelines ● Git repositories ● Branching strategies Infrastructure Deployment ● Infrastructure as Code ● ARM Templates ● Bicep ● Terraform Automation ● Deployment automation ● Environment promotion ● Release management 5. Monitoring and Observability Azure Monitor ● Monitoring strategy implementation ● Alert configuration ● Dashboard creation Log Analytics ● KQL query development ● Performance analysis ● Troubleshooting Application Monitoring ● Azure Application Insights ● Distributed tracing ● Metrics collection AKS Monitoring ● Cluster health monitoring ● Pod performance monitoring ● Resource optimization6. Azure Event Hub Candidates must have strong hands-on experience in: ● Event Hub architecture ● Namespace management ● Consumer groups ● Scaling and throughput units ● Monitoring and troubleshooting ● Security and access control 7. Azure Functions Candidates must have hands-on experience in: ● Deployment and operations support ● Function App management ● Scaling configuration ● Managed Identity integration ● Monitoring and troubleshooting ● CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in: ● NiFi cluster setup and maintenance ● Flow deployment ● Security configuration ● Performance tuning ● Monitoring ● Troubleshooting Preferred ● NiFi Registry ● Cluster scaling ● High availability implementation 9. Infrastructure as Code Must-Have Skills ● Terraform ● Bicep ● ARM Templates Required Capabilities ● Platform provisioning ● Environment consistency ● Version-controlled infrastructure Good-to-Have Skills Platform Engineering ● GitOps using ArgoCD or Flux ● Platform automation ● Self-service platform concepts ● Internal Developer Platform Observability ● Grafana ● Prometheus ● OpenTelemetry Security ● Microsoft Defender for Cloud ● Kubernetes security best practices ● Container security ● Secrets management ● Vulnerability management Data Platform ● Azure Data Factory ● Synapse Analytics Cloud ● Multi-region architecture ● Disaster recovery planning ● Landing Zone implementation Scripting ● PowerShell ● Bash ● Python Key Responsibilities ● Manage and support AKS-based Big Data platform infrastructure. ● Administer Apache NiFi, Azure Event Hub, and Azure Functions. ● Implement and maintain CI/CD pipelines using Azure DevOps. ● Monitor platform health using Azure Monitor and Log Analytics. ● Troubleshoot platform, networking, and infrastructure issues. ● Implement infrastructure automation through Terraform and Bicep. ● Ensure platform security, governance, and compliance. ● Collaborate with Data Engineering teams for platform enablement. ● Perform upgrades, patching, capacity planning, and optimization. ● Support production incidents and platform reliability initiatives.

Experience Requirements

● 8+ years of overall IT experience ● 5+ years of experience in Cloud Infrastructure Engineering ● 4+ years of experience working with Kubernetes platforms ● 3+ years of experience supporting enterprise-scale Data Platforms ● 3+ years of experience in DevOps/SRE practices ● 2+ years of experience working with Azure Data Services

Mandatory Skills

Candidates must possess strong hands-on experience in the following areas: ● Apache NiFi ● Azure Event Hub ● Azure Functions ● Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered.

Requirements

1. Azure Kubernetes Service – AKS

Strong Expertise

● AKS cluster deployment and administration ● Cluster upgrades and patch management ● Node pool management ● Ingress Controllers, including NGINX and Application Gateway ● RBAC implementation ● AKS security hardening ● Autoscaling using HPA, VPA, and Cluster Autoscaler ● Persistent storage management ● Troubleshooting cluster issues ● Network policies

Experience

● Production support of AKS environments ● Multi-node and multi-environment AKS setups ● High availability and disaster recovery implementation

2. Kubernetes and Azure Networking

Kubernetes Networking ● Kubernetes networking concepts ● Services, Ingress, and Load Balancers ● DNS resolution ● Pod-to-Pod communication ● Pod-to-Service communication ● Network Policies ● Service Mesh concepts, preferably Istio ● CNI Architecture

Azure Networking

● Virtual Networks ● Subnets ● Private Endpoints ● Peering ● Network Security Groups ● User Defined Routes ● Azure Firewall ● VPN connectivity

3. Docker and Containerization

● Docker image creation and optimization ● Container lifecycle management ● Registry management ● Security scanning ● Multi-stage builds ● Container troubleshooting ● Azure Container Registry ● Image versioning and governance

4. Azure DevOps and CI/CD

CI/CD Expertise ● Azure DevOps Pipelines ● YAML pipelines ● Build and Release pipelines ● Git repositories ● Branching strategies

Infrastructure Deployment

● Infrastructure as Code ● ARM Templates ● Bicep ● Terraform

Automation

● Deployment automation ● Environment promotion ● Release management

5. Monitoring and Observability

Azure Monitor

● Monitoring strategy implementation ● Alert configuration ● Dashboard creation

Log Analytics

● KQL query development ● Performance analysis ● Troubleshooting

Application Monitoring

● Azure Application Insights ● Distributed tracing ● Metrics collection

AKS Monitoring

● Cluster health monitoring ● Pod performance monitoring ● Resource optimization

6. Azure Event Hub Candidates must have strong hands-on experience in: ● Event Hub architecture ● Namespace management ● Consumer groups ● Scaling and throughput units ● Monitoring and troubleshooting ● Security and access control 7. Azure Functions Candidates must have hands-on experience in: ● Deployment and operations support ● Function App management ● Scaling configuration ● Managed Identity integration ● Monitoring and troubleshooting ● CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in: ● NiFi cluster setup and maintenance ● Flow deployment ● Security configuration ● Performance tuning ● Monitoring ● Troubleshooting Preferred ● NiFi Registry ● Cluster scaling ● High availability implementation 9. Infrastructure as Code Must-Have Skills ● Terraform ● Bicep ● ARM Templates

Required Capabilities

● Platform provisioning ● Environment consistency ● Version-controlled infrastructure Good-to-Have Skills Platform Engineering ● GitOps using ArgoCD or Flux ● Platform automation ● Self-service platform concepts ● Internal Developer Platform Observability ● Grafana ● Prometheus ● OpenTelemetry Security ● Microsoft Defender for Cloud

● Kubernetes security best practices ● Container security ● Secrets management ● Vulnerability management Data Platform ● Azure Data Factory ● Synapse Analytics Cloud ● Multi-region architecture ● Disaster recovery planning ● Landing Zone implementation Scripting ● PowerShell ● Bash ● Python

Key Responsibilities

● Manage and support AKS-based Big Data platform infrastructure. ● Administer Apache NiFi, Azure Event Hub, and Azure Functions. ● Implement and maintain CI/CD pipelines using Azure DevOps. ● Monitor platform health using Azure Monitor and Log Analytics. ● Troubleshoot platform, networking, and infrastructure issues. ● Implement infrastructure automation through Terraform and Bicep. ● Ensure platform security, governance, and compliance. ● Collaborate with Data Engineering teams for platform enablement. ● Perform upgrades, patching, capacity planning, and optimization. ● Support production incidents and platform reliability initiatives.

Two95 International Inc., is a global technology firm specializing in enterprise solutions that evolves over BPM, Mobility, Cloud, Analytics, E-commerce & Social Business. Our client base includes several Fortune 500 and mid-market companies across industries and varying geographies.With vast knowledge and knowhow of 20 years in the IT field, we have been chosen as INC500 fastest growing company in North America in 2013. With the accolade of being ranked 11th in Human Resources by INC500, we have also been nominated as the 3rd fastest growing company in South Jersey by SJBM. We are ranked among the Top 20 IT Companies in New Jersey based on the year-on-year growth for the last 3 years. With a seasoned team of highly qualified personnel, our offices are located in New Jersey, Canada and India.Our Specialties Direct Hire, Contingent Staffing, Managed Outsourced Services..

Two95 International Inc., is a global technology firm specializing in enterprise solutions that evolves over BPM, Mobility, Cloud, Analytics, E-commerce & Social Business. Our client base includes several Fortune 500 and mid-market companies across industries and varying geographies.

With vast knowledge and knowhow of 20 years in the IT field, we have been chosen as INC500 fastest growing company in North America in 2013. With the accolade of being ranked 11th in Human Resources by INC500, we have also been nominated as the 3rd fastest growing company in South Jersey by SJBM. We are ranked among the Top 20 IT Companies in New Jersey based on the year-on-year growth for the last 3 years. With a seasoned team of highly qualified personnel, our offices are located in New Jersey, Canada and India.

Our Specialties

Direct Hire, Contingent Staffing, Managed Outsourced Services..