Responsibility
Job Title: Databricks Subject Matter Expert (SME)
Experience: 12–15 Years
Employment Type: Contract / Freelance
Location: Noida / Hybrid / Remote
Role Summary
We are looking for an experienced Databricks Subject Matter Expert (SME) to lead the architecture, development, optimization, and governance of enterprise-scale data platforms. The ideal candidate will have strong expertise in Databricks Lakehouse, Apache Spark, Delta Lake, cloud platforms, and modern data engineering practices. This role involves providing technical leadership, defining best practices, mentoring teams, and supporting data, analytics, and AI/ML initiatives.
Key Responsibilities
- Design scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform
- Build and optimize ETL/ELT pipelines using Apache Spark, PySpark, and Delta Lake
- Design and manage Medallion Architecture (Bronze, Silver, and Gold layers)
- Optimize Spark workloads for performance, scalability, reliability, and cost efficiency
- Implement data governance and security using Unity Catalog, RBAC, data lineage, and auditing
- Integrate Databricks with Azure, AWS, or GCP, along with data warehouses, BI tools, and external systems
- Develop and support CI/CD pipelines and Infrastructure-as-Code using Terraform or similar tools
- Support streaming, AI/ML, MLOps, and Generative AI workloads
- Lead architecture reviews, troubleshoot production issues, and establish enterprise best practices
- Mentor technical teams and collaborate with business stakeholders, architects, and engineering teams
Required Skills & Experience
- 12–15 years of experience in Data Engineering, Data Architecture, or Data Platforms
- Strong hands-on expertise in Databricks Lakehouse
- Advanced experience with Apache Spark, PySpark, Scala, SQL, and Python
- Strong knowledge of Delta Lake, Unity Catalog, and Databricks Workflows
- Experience with Azure Databricks, AWS Databricks, or GCP Databricks
- Expertise in large-scale data processing, data modeling, and ETL/ELT development
- Experience with Kafka, Spark Streaming, or other streaming technologies
- Knowledge of CI/CD, Git, Azure DevOps, GitHub Actions, or Jenkins
- Strong understanding of data governance, cloud security, compliance, and access controls
- Excellent technical leadership, stakeholder management, and communication skills
Preferred Skills
- Databricks Certified Data Engineer Professional
- Experience with MLflow, MLOps, and Generative AI solutions
- Knowledge of Terraform and cloud automation
- Exposure to Power BI, Tableau, Snowflake, or Azure Synapse
- Experience mentoring teams and leading enterprise data modernization initiatives
Education
Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, Engineering, or a related field.