Responsibility
Job Title: Data Engineer – Microsoft Fabric
Experience: 8+ Years
Employment Type: Contract (3 Months + Extendable)
Work Mode: Remote
Work Timings: 1:30 PM–10:30 PM IST
Role Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Microsoft Fabric and PySpark to design, develop, and maintain scalable, production-grade data solutions.
The ideal candidate should have extensive experience building Microsoft Fabric Lakehouse solutions, implementing Medallion Architecture (Bronze, Silver, and Gold layers), and developing reusable, metadata-driven data ingestion frameworks. Strong knowledge of PySpark, SQL, Azure DevOps, CI/CD, data engineering best practices, and Fabric deployment strategies is essential.
Key Responsibilities
- Analyze source systems and business requirements and translate them into scalable data models, pipeline designs, and end-to-end data solutions.
- Design and develop reusable data ingestion frameworks in Microsoft Fabric using PySpark notebooks, supporting data ingestion from databases, flat files, APIs, and other source systems.
- Build and maintain scalable ETL/ELT pipelines across the Bronze, Silver, and Gold layers using Lakehouse and Medallion Architecture principles.
- Ingest and manage raw data in the Bronze layer while ensuring data reliability, scalability, traceability, and efficient incremental or delta processing.
- Transform raw data into clean, standardized, and validated Silver-layer datasets through deduplication, conformance, validation, and data-quality checks.
- Design and develop Gold-layer data models, including fact tables, dimension tables, aggregates, semantic models, and reporting-ready datasets.
- Develop metadata-driven and configurable ingestion frameworks to support reusable, scalable, and maintainable data pipelines.
- Implement data-quality and validation rules covering completeness, accuracy, freshness, consistency, and reconciliation.
- Build robust error-handling and recovery mechanisms, including logging, exception handling, retry logic, checkpointing, alerting, idempotent processing, and failure recovery.
- Monitor pipeline performance, proactively identify failures and bottlenecks, and optimize data-processing workloads for performance and cost efficiency.
- Implement CI/CD and DevOps practices using Azure DevOps to support deployment across Development, Test, and Production environments.
- Manage Microsoft Fabric workspaces, Lakehouse maintenance, deployment processes, and secure access to credentials using Azure Key Vault.
- Collaborate with business, BI, data architecture, and engineering teams to deliver end-to-end data solutions.
Required Skills & Experience
- 8+ years of experience in Data Engineering, ETL/ELT Development, or related roles.
- Extensive hands-on experience with Microsoft Fabric, including Fabric Lakehouse architecture, workspace management, and production deployments.
- Strong expertise in PySpark, including writing, debugging, optimizing, and maintaining production-grade data-processing code.
- Strong understanding and practical implementation experience with: Microsoft Fabric, Lakehouse Architecture, Medallion Architecture, Bronze, Silver, and Gold data layers, PySpark Notebooks, ETL/ELT Pipelines, Batch and Incremental/Delta Data Loads, Metadata-Driven Data Ingestion Frameworks
- Experience ingesting data from: Relational databases, Database tables, Flat files, APIs, Multiple enterprise data sources
- Strong SQL skills, including writing, modifying, debugging, and optimizing complex queries.
- Experience designing fact and dimension tables, aggregates, semantic models, and analytics-ready datasets.
- Strong knowledge of data quality, validation, reconciliation, and data lineage.
- Experience implementing scalable and reusable data pipeline frameworks.
- Strong understanding of Spark fundamentals, including distributed processing, partitioning, optimization, and performance tuning.
- Experience with Azure DevOps, Git, CI/CD pipelines, and multi-environment deployments.
- Knowledge of software design standards and data engineering best practices.
Preferred Skills
- Experience with Azure Data Factory (ADF) and other Azure data services.
- Experience with OneLake, Microsoft Fabric Data Factory, or Fabric Data Pipelines.
- Knowledge of Microsoft Fabric capacity management and cost optimization.
- Experience with data governance, security, and enterprise data architecture.
- Exposure to Power BI semantic models and reporting solutions.
- Experience implementing automated monitoring and alerting for enterprise data platforms.