Databricks Platform & Infrastructure Engineer

Responsibility

Job Title: Databricks Platform & Infrastructure Engineer (Unity Catalog Enablement)

Experience: 3–6 Years
Employment Type: Contract
Work Mode: Remote
Work Timings: 1:30 PM to 10:30 PM

Role Overview

We are looking for a Databricks Platform & Infrastructure Engineer to deploy, configure, and support Azure and Databricks platform capabilities required to enable Unity Catalog and establish a secure, scalable, and governed data platform.

This role is implementation-focused and involves working from architect-defined designs to provision infrastructure, configure networking and identity, enable CI/CD deployments, and support BAU operational readiness for Unity Catalog.

Key Responsibilities

Azure Platform & Infrastructure
  • Configure and manage Azure subscriptions, resource groups, and environment structures.
  • Implement Infrastructure as Code (IaC) using Terraform.
  • Configure Azure DevOps projects, repositories, and deployment pipelines.
  • Deploy and manage ADLS Gen2 storage accounts and supporting Azure services.
  • Implement environment separation across Development, Test, and Production.
  • Support platform monitoring, diagnostic logging, alerting, and operational dashboards.
Network Security & Identity
  • Configure Microsoft Entra ID integration with Databricks and Unity Catalog.
  • Implement RBAC and ABAC access control models.
  • Configure Managed Identities, Service Principals, and authentication mechanisms.
  • Configure Databricks networking, including VNet injection, Secure Cluster Connectivity, private connectivity patterns, and Access Connectors.
  • Support Hub-and-Spoke networking patterns and enterprise security standards.
  • Enable secure connectivity to ADLS, source systems, and enterprise platforms.
  • Work with Infrastructure, Security, and Architecture teams to obtain approvals and align with organizational standards.
Databricks Platform & Unity Catalog
  • Deploy and configure Databricks accounts and workspaces.
  • Configure Unity Catalog metastore, catalogs, schemas, and storage locations.
  • Configure storage credentials, external locations, and governance controls.
  • Implement environment isolation and workspace administration standards.
  • Support Hive Metastore-to-Unity Catalog migration activities.
  • Implement platform security, governance, and access management best practices.
CI/CD & Automation
  • Develop and maintain Infrastructure-as-Code templates and modules.
  • Implement automated deployment pipelines using Azure DevOps.
  • Establish branching, pull request, release management, and deployment governance standards.
  • Implement secure credential management using Azure Key Vault, Managed Identities, and Databricks secret management capabilities.
  • Support source control, release management, and deployment standards.
  • Develop operational documentation and deployment runbooks.
Governance & Operating Model
  • Implement Unity Catalog RBAC and ABAC access models in line with the agreed security design.
  • Configure catalog, schema, storage credential, and external location ownership.
  • Support the definition and implementation of UC admin groups, data owner groups, and platform support roles.
  • Configure governed access patterns for BAU teams using approved Microsoft Entra ID groups.
  • Support access request, approval, audit, and recertification processes.
  • Configure monitoring and audit logging to support BAU operations and security reviews.
  • Produce UC administration runbooks covering ownership, access changes, onboarding, and support processes.
  • Enable Unity Catalog audit, lineage, and access governance capabilities to support regulatory and operational requirements.
  • Implement cost monitoring, tagging standards, and usage optimization controls across Databricks and Azure resources.

Must-Have Skills

Azure & Infrastructure
  • Azure Subscriptions and Resource Groups
  • Azure DevOps
  • ADLS Gen2
  • Azure Networking and Security
  • Microsoft Entra ID
  • Terraform for Azure infrastructure deployment
  • Databricks Asset Bundles (DABs) for Databricks object deployment and environment promotion
Databricks
  • Databricks Workspace Administration
  • Unity Catalog
  • Databricks Account Administration
  • Storage Credentials and External Locations
  • Hive Metastore-to-Unity Catalog Migration
  • Databricks Security and Governance
Security & Governance
  • RBAC and ABAC access models
  • Managed Identities and Service Principals
  • Private Networking and Private Endpoints
  • Data Governance and Access Management

Team Collaboration & Soft Skills

  • Strong collaboration skills, working across Infrastructure, Security, Data Engineering, and Architecture teams.
  • Ability to facilitate technical workshops and governance discussions with client stakeholders.
  • Strong communication skills with the ability to explain complex technical concepts to both technical and non-technical audiences.
  • Proactive approach to identifying risks, dependencies, and operational challenges.
  • Ability to work effectively within distributed onshore and offshore teams.
  • Strong documentation and knowledge-sharing capabilities.
  • Outcome-focused mindset with a commitment to delivering secure, scalable, and operationally sustainable solutions.

Nice-to-Have Skills

  • Databricks Asset Bundles (DABs)
  • Azure Key Vault
  • Azure Private DNS
  • Microsoft Purview
  • Data Governance Frameworks
  • Delta Live Tables (DLT)
  • FinOps and Cloud Cost Optimization
  • Data Vault and Lakehouse Architecture
  • Experience supporting enterprise-scale Data Platform Modernization or Unity Catalog enablement

Apply Now

    91 906 888 2169
    Contact Us