Job Description
We are seeking a Senior DevOps / MLOps / Data Engineer with strong experience in Azure to lead platform engineering, deployment automation, and AI/ML model deployment. This role focuses on building scalable, secure, and cost-efficient cloud solutions while enabling end-to-end ML lifecycle management and data engineering capabilities.
Key Responsibilities
DevOps, Deployment & MLOps (Primary Focus)
- Design and implement CI/CD pipelines using Azure DevOps, Git, and YAML
- Manage end-to-end deployments across Dev, QA, and Production environments
- Lead deployment of AI/ML models into production using automated pipelines
- Implement model lifecycle management (training, validation, deployment, monitoring)
- Deploy and manage batch and real-time inference models
- Automate infrastructure provisioning using ARM, Bicep, or Terraform
- Manage Azure resources via Portal, CLI, and scripting
- Oversee Databricks cluster management, scaling, and performance tuning
- Implement release strategies, versioning, rollback, and environment configuration
- Build scalable API-based model inference endpoints
- Integrate Azure AI services (e.g., Azure AI Search, Azure AI Foundry)
- Monitor system reliability, model performance, and drift
Data Engineering
- Design and develop data pipelines using Azure Data Factory, Databricks, and ADLS
- Build and optimize data models in Azure SQL, SQL Server, and Oracle
- Implement ETL/ELT processes for large-scale data processing
- Ensure data quality, governance, and performance optimization
- Support medallion architecture (Bronze, Silver, Gold layers)
Security & Compliance
- Implement secure cloud architecture using RBAC, Managed Identities, and Key Vault
- Secure data pipelines and ML endpoints (encryption, private endpoints, network controls)
- Ensure compliance with data protection and governance standards
- Manage secrets, credentials, and access policies
Cost Optimization
- Optimize cloud costs across Databricks, storage, and compute resources
- Implement cluster right-sizing and auto-scaling strategies
- Monitor usage and enforce cost governance
- Recommend cost-performance improvements
API & Integration
- Design and build scalable REST APIs for data access and model inference
- Develop API-based integrations with internal and external systems
- Enable real-time and batch data integrations
- Implement API security (authentication, throttling, versioning)
- Support event-driven architectures and messaging systems
Required Qualifications
- 10+ years of IT experience, including 5+ years in DevOps / MLOps / Data Engineering
- Strong expertise in:
- Azure ecosystem (ADF, Databricks, ADLS, Azure SQL)
- CI/CD pipelines, Git, and YAML
- Infrastructure as Code (ARM, Bicep, Terraform)
- SQL and relational databases (Azure SQL, Oracle)
- REST API development and integration
- Proven experience in:
- Deploying AI/ML models into production
- End-to-end deployment pipelines and release management
- Cluster management and optimization
Nice to Have
- Azure AI Search and Azure AI Foundry
- Event-driven architecture (Event Grid, Service Bus)
- Streaming platforms (Kafka, Event Hubs)
- Containerization (Docker, Kubernetes, AKS)
- Experience with LLMs / Generative AI pipelines
- Data governance and medallion architecture
Soft Skills
- Strong problem-solving and troubleshooting skills
- Ability to collaborate across DevOps, Data, and ML teams
- Excellent communication and documentation
- Leadership and mentoring capabilities
Must-Have Requirements
- 10+ years designing and implementing CI/CD pipelines (Azure DevOps, Git, YAML)
- 10+ years deploying AI/ML models using automated pipelines
- 10+ years implementing ML lifecycle management
- 10+ years building data pipelines using ADF, Databricks, and ADLS
Job Tags