Data Engineer

Job Description

Role: Data Engineer Location: Bangalore Type: Fulltime Experience: 6 to 10 yrs Notice: Immediate Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL) Role Overview: Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog, Slowly Changing Dimension Type 2 (SCD2), and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing data pipelines and ETL workflows while ensuring data integrity, scalability, and security within the Azure ecosystem. Key Responsibilities: Develop and optimize data pipelines using Azure Data Factory (ADF) and Azure Databricks for large-scale data processing. Implement Slowly Changing Dimension in Delta Tables to manage historical data changes effectively. Leverage Unity Catalog for secure and organized data governance, cataloging, and access control across Databricks. Write efficient PySpark code to process and transform large datasets, ensuring high performance and scalability. Design and implement ETL/ELT solutions to integrate data from multiple sources into Delta Lake. Monitor, debug, and optimize existing data pipelines to ensure smooth operations and minimal downtime. Ensure data quality, consistency, and lineage tracking through best practices and automation. Collaborate with data architects, analysts, and business teams to define requirements and implement data-driven solutions. Required Skills & Qualifications: 6+ years of experience in Data Engineering with a focus on Azure technologies. Expertise in Azure Data Factory (ADF) & Azure Databricks for ETL/ELT workflows. Strong knowledge of Delta Tables & Unity Catalog for efficient data storage and management. Experience with Slowly Changing Dimensions (SCD2) implementation in Delta Lake. Proficiency in PySpark for large-scale data processing & transformation. Hands-on experience with SQL & performance tuning for data pipelines. Understanding of data governance, security, and compliance best practices in Azure. Knowledge of CI/CD, DevOps practices for data pipeline automation. Preferred Qualifications: Experience with Azure Synapse Analytics, Data Lakes, and Power BI integration . Knowledge of Kafka or Event Hub for real-time data ingestion. Certifications in Microsoft Azure (DP-203, DP-900) or Databricks are a plus. Show more Show less