Senior Data Engineer

Job Description

OverviewWe are seeking a highly skilled Senior Data Engineer to lead the design, development, and optimization of large-scale data infrastructure, pipelines, and platforms. This role requires expertise across the full spectrum of data engineering, including cloud and on-premise systems, real-time streaming, orchestration frameworks, and robust data governance practices.You will be a key contributor in shaping the organization’s data ecosystem, enabling analytics, machine learning, and real-time decision-making at scale. This position demands not just competence—but mastery—of the core disciplines and tools in modern data engineering. ResponsibilitiesData Platform ArchitectureDesign, implement, and manage hybrid data platforms across on-premise and cloud environments.Build scalable and reliable data lake and warehouse solutions using best-in-class storage formats and compute frameworks.Collaborate with cross-functional teams to define data architecture and technology strategy aligned with business objectives.Ingestion and OrchestrationDevelop and maintain robust data ingestion workflows using tools such as Apache NiFi, Kafka, and custom connectors.Implement data orchestration pipelines with tools like Dagster, Airflow, or Prefect for both batch and streaming data.Build modular, maintainable workflows that adhere to best practices in monitoring, error handling, and retries.ETL/ELT Pipeline DevelopmentDesign and optimize ETL/ELT pipelines that process data at scale from multiple systems into analytical environments.Ensure data workflows are highly performant, idempotent, and compliant with SLAs.Use Spark, dbt, or custom code to transform, enrich, and validate data.Data Modeling and WarehousingCreate and maintain normalized and denormalized schemas for analytical workloads using star and snowflake models.Work with cloud and on-premise databases and warehouses including PostgreSQL, Redshift, BigQuery, Snowflake, and Hive.Define partitioning, bucketing, and indexing strategies to ensure query efficiency.Infrastructure and DevOpsDeploy and maintain infrastructure using Terraform, Ansible, or shell scripting for both cloud and on-premise systems.Implement CI/CD pipelines for data services using Jenkins, GitLab CI, or similar tools.Utilize Docker and optionally Kubernetes to package and manage data applications.Data Governance and QualityDefine and enforce data quality policies using tools like Great Expectations or Deequ.Establish lineage and metadata tracking through solutions like Apache Atlas, Amundsen, or Collibra.Implement access control, encryption, and audit policies to ensure data security and compliance.Monitoring and OptimizationMonitor pipeline health, job performance, and system metrics using Prometheus, Grafana, or ELK.Continuously optimize workflows and queries to minimize cost and latency.Perform root cause analysis and troubleshooting for data issues in production systems.Collaboration and LeadershipMentor junior and mid-level data engineers, participate in technical reviews, and help define team standards.Work closely with data scientists, analysts, software engineers, and product managers to gather requirements and translate them into robust data solutions.Promote a culture of high quality, documentation, reusability, and operational excellence. QualificationsRequiredBachelor’s or Master’s degree in Computer Science, Engineering, or related field.At least 5 years of experience as a data engineer, with expertise in both on-premise and cloud environments.Deep experience with Apache NiFi, Dagster, and orchestration frameworks such as Airflow or Prefect.Proficiency in Python, SQL, and optionally Scala or Java.Strong understanding of distributed systems, including Hadoop, Spark, and Kafka.Demonstrated experience building secure, scalable, and maintainable data pipelines and infrastructure.Familiarity with modern data stack tools and infrastructure automation.PreferredExperience with real-time data processing and CDC pipelines.Exposure to regulatory and high-security environments (e.g., healthcare, finance, industrial systems).Certifications in AWS, GCP, or Azure for data engineering or analytics.Contributions to open-source data tooling or internal platform development. What We OfferA high-impact role in a data-driven organization focused on innovation and scalability.Flexible working environment and strong support for personal development.Competitive compensation and benefits, including performance-based incentives.Opportunities to work on high-visibility projects and influence enterprise data architecture. How to Apply Please share your updated CV along with the following details:Current CTCExpected CTCNotice Period Email to: jignesh.pandoriya@merillife.com