Senior Data Engineer

Job Description

Designation: Senior Data Engineer Preferred Experience: 7+ years Responsibilities: Design, develop, and maintain data pipelines for ingesting, processing, and transforming data from various sources into actionable insights. Integrate data from disparate sources (databases, APIs, and files) into a unified data platform using ETL processes and data integration techniques. Design and implement data models, schemas, and data structures to support analytical queries, reporting, and business intelligence needs. Optimize database performance, query execution, and data processing workflows for efficiency, scalability, and reliability. Ensure data quality, integrity, and consistency through data validation, cleansing, deduplication, and error-handling mechanisms. Architect and implement data solutions on Azure cloud platforms, leveraging Azure services for data storage, processing, and analytics. Implement data security measures, encryption techniques, and access controls to protect sensitive data and ensure compliance with regulations (e.g., GDPR, HIPAA). Work closely with cross-functional teams, data scientists, analysts, and stakeholders to understand data requirements, provide data solutions, and communicate insights effectively. Document data processes, workflows, and best practices, and promote data governance standards, data lineage, and metadata management. Stay updated with emerging technologies, industry trends, and best practices in data engineering, cloud computing, and data analytics to drive innovation and continuous improvement. Required Skills: Proficiency in writing complex SQL queries, stored procedures, and functions for data extraction, transformation, and analysis. Experience in database design, optimization, and management using SQL Server/Azure SQL Database. Knowledge of data modeling techniques, including entity-relationship diagrams, dimensional modeling, and data normalization, is needed to design efficient data structures. Familiarity with ETL (Extract, Transform, Load) processes and tools such as Azure Data Factory, SSIS (SQL Server Integration Services), or other data integration platforms. Hands-on experience with Azure cloud services, including Azure SQL Database, Azure Data Factory and Azure Storage. Experience handling large-scale data processing and analytics with strong analytical skills and attention to detail. Strong statistical knowledge. Excellent communication skills. Good To Have Skills: Experience working with C#, .Net Framework Experience with other Azure cloud services and sharepoint Experience with Python as programming language Experience manipulating unstructured data with regular expressions. Basic understanding of machine learning concepts and algorithms for data mining, predictive modeling, and statistical analysis. Knowledge of data warehousing concepts, methodologies, and tools for building and maintaining data warehouses or data marts.