Responsibility
Job Position: Senior Databricks / Azure Data Engineer
Experience: 10+ Years
Location: Remote
Contract Duration: 3 Months + Extendable
Shift Timings: 1:30 PM to 10:30 PM IST
Role Summary
We are looking for a Senior Databricks / Azure Data Engineer with strong and recent hands-on experience in Databricks implementation projects. The ideal candidate should have expertise in modern Databricks capabilities and strong end-to-end data engineering experience across data processing, modelling, governance, and cloud data platforms.
Priority will be given to candidates whose most recent project is focused on Databricks. Candidates with recent Fabric, Azure, or Snowflake experience should not be prioritized unless they also have strong, current hands-on Databricks implementation experience.
Key Responsibilities
- Design, develop, and implement end-to-end data engineering solutions using Azure Databricks, including modern Databricks capabilities such as Lakeflow Declarative Pipelines and Unity Catalog.
- Develop scalable data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake.
- Implement and manage Unity Catalog, including migration from legacy Databricks workspaces.
- Design and implement Data Vault models and apply strong data modelling practices across enterprise data platforms.
- Build and maintain data pipelines using Azure Data Factory, ADLS, Azure Synapse, and Azure Blob Storage.
- Work with ETL/ELT, Kafka, Hadoop, and other data engineering technologies to support large-scale data processing.
- Implement data governance, data quality, security, and access controls using Entra ID/Azure AD, Key Vault, RBAC, and secure connectivity.
- Optimize Databricks workloads and support performance improvements across Spark and Delta Lake environments.
- Collaborate with cross-functional teams on architecture, implementation, migration, deployment, and production support.
Mandatory Skills & Experience
- 10+ years of overall Data Engineering experience.
- Strong and recent hands-on experience with Azure Databricks.
- Recent end-to-end Databricks implementation experience.
- Hands-on experience with Lakeflow Declarative Pipelines.
- Strong experience with Unity Catalog, including migration from legacy workspaces.
- Strong knowledge of Data Vault modelling and data modelling concepts.
- Excellent hands-on skills in Python, PySpark, and SQL.
- Experience with Azure Data Factory, ADLS, Azure Synapse, and Azure Blob Storage.
- Strong knowledge of ETL/ELT, Apache Spark, Kafka, and Hadoop.
- Experience with data governance and data quality.
- Understanding of cloud security concepts including Entra ID/Azure AD, Key Vault, RBAC, and secure connectivity.
- Experience with on-premises infrastructure/clusters such as Cloudera or Hortonworks is an advantage.
Preferred Skills
- Databricks performance optimization and strong experience with Delta Lake.
- Experience with Airflow, Informatica, Talend, or NiFi.
- Experience with Azure DevOps, CI/CD, and Terraform.
- Exposure to enterprise-scale Databricks migration and modernization initiatives.
Key Skills
Azure Databricks | Lakeflow | Unity Catalog | Data Vault | PySpark | Python | SQL | Apache Spark | ADF | ADLS | Synapse | Delta Lake | ETL/ELT | Kafka | Data Engineering