Big Data Platform Engineer

Location: India

Workplace: remote

Employment Type: full

Description

Experience Requirements

● 8+ years of overall IT experience ● 5+ years of experience in Cloud Infrastructure Engineering ● 4+ years of experience working with Kubernetes platforms ● 3+ years of experience supporting enterprise-scale Data Platforms ● 3+ years of experience in DevOps/SRE practices ● 2+ years of experience working with Azure Data Services

Mandatory Skills

Candidates must possess strong hands-on experience in the following areas: ● Apache NiFi ● Azure Event Hub ● Azure Functions ● Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered.

Requirements

1. Azure Kubernetes Service – AKS

Strong Expertise

● AKS cluster deployment and administration ● Cluster upgrades and patch management ● Node pool management ● Ingress Controllers, including NGINX and Application Gateway ● RBAC implementation ● AKS security hardening ● Autoscaling using HPA, VPA, and Cluster Autoscaler ● Persistent storage management ● Troubleshooting cluster issues ● Network policies

Experience

● Production support of AKS environments ● Multi-node and multi-environment AKS setups ● High availability and disaster recovery implementation

2. Kubernetes and Azure Networking

Kubernetes Networking ● Kubernetes networking concepts ● Services, Ingress, and Load Balancers ● DNS resolution ● Pod-to-Pod communication ● Pod-to-Service communication ● Network Policies ● Service Mesh concepts, preferably Istio ● CNI Architecture

Azure Networking

● Virtual Networks ● Subnets ● Private Endpoints ● Peering ● Network Security Groups ● User Defined Routes ● Azure Firewall ● VPN connectivity

3. Docker and Containerization

● Docker image creation and optimization ● Container lifecycle management ● Registry management ● Security scanning ● Multi-stage builds ● Container troubleshooting ● Azure Container Registry ● Image versioning and governance

4. Azure DevOps and CI/CD

CI/CD Expertise ● Azure DevOps Pipelines ● YAML pipelines ● Build and Release pipelines ● Git repositories ● Branching strategies

Infrastructure Deployment

● Infrastructure as Code ● ARM Templates ● Bicep ● Terraform

Automation

● Deployment automation ● Environment promotion ● Release management

5. Monitoring and Observability

Azure Monitor

● Monitoring strategy implementation ● Alert configuration ● Dashboard creation

Log Analytics

● KQL query development ● Performance analysis ● Troubleshooting

Application Monitoring

● Azure Application Insights ● Distributed tracing ● Metrics collection

AKS Monitoring

● Cluster health monitoring ● Pod performance monitoring ● Resource optimization

6. Azure Event Hub Candidates must have strong hands-on experience in: ● Event Hub architecture ● Namespace management ● Consumer groups ● Scaling and throughput units ● Monitoring and troubleshooting ● Security and access control 7. Azure Functions Candidates must have hands-on experience in: ● Deployment and operations support ● Function App management ● Scaling configuration ● Managed Identity integration ● Monitoring and troubleshooting ● CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in: ● NiFi cluster setup and maintenance ● Flow deployment ● Security configuration ● Performance tuning ● Monitoring ● Troubleshooting Preferred ● NiFi Registry ● Cluster scaling ● High availability implementation 9. Infrastructure as Code Must-Have Skills ● Terraform ● Bicep ● ARM Templates

Required Capabilities

● Platform provisioning ● Environment consistency ● Version-controlled infrastructure Good-to-Have Skills Platform Engineering ● GitOps using ArgoCD or Flux ● Platform automation ● Self-service platform concepts ● Internal Developer Platform Observability ● Grafana ● Prometheus ● OpenTelemetry Security ● Microsoft Defender for Cloud

● Kubernetes security best practices ● Container security ● Secrets management ● Vulnerability management Data Platform ● Azure Data Factory ● Synapse Analytics Cloud ● Multi-region architecture ● Disaster recovery planning ● Landing Zone implementation Scripting ● PowerShell ● Bash ● Python

Key Responsibilities

● Manage and support AKS-based Big Data platform infrastructure. ● Administer Apache NiFi, Azure Event Hub, and Azure Functions. ● Implement and maintain CI/CD pipelines using Azure DevOps. ● Monitor platform health using Azure Monitor and Log Analytics. ● Troubleshoot platform, networking, and infrastructure issues. ● Implement infrastructure automation through Terraform and Bicep. ● Ensure platform security, governance, and compliance. ● Collaborate with Data Engineering teams for platform enablement. ● Perform upgrades, patching, capacity planning, and optimization. ● Support production incidents and platform reliability initiatives.