Data Engineer

Job Description

Job Title: Senior Data Engineer Apache Airflow Position Overview: We are seeking a highly skilled and experienced Data Engineer specializing in Apache Airflow to design, develop, and optimize scalable data pipelines and workflows . In this role, you will be responsible for automating end-to-end data processing, ensuring high availability, and optimizing performance . You will work closely with data engineers, data scientists, and stakeholders to build robust, cloud-based data solutions . Key Responsibilities: 1. Design & Development Lead the design, development, and optimization of scalable data pipelines and workflows using Apache Airflow . Create reusable components and efficient Directed Acyclic Graphs (DAGs) for data processing. 2. Workflow Automation & Orchestration Automate batch and real-time data pipelines using Airflow , integrating multiple data sources and transformations . Ensure fault-tolerant, high-availability workflows with error handling and retry mechanisms . 3. Cloud Platform Integration Develop, deploy, and manage scalable Airflow DAGs in cloud environments (AWS preferred). Utilize AWS services such as S3, EC2, Lambda, and RDS for seamless cloud-based data integration . 4. Performance Optimization & Monitoring Monitor, analyze, and optimize pipeline performance to meet SLAs . Troubleshoot and resolve workflow bottlenecks, task failures, and execution delays proactively. 5. Collaboration & Mentorship Work closely with data engineers, data scientists, and business stakeholders to align solutions with business needs . Mentor junior engineers , providing guidance on Airflow best practices and workflow efficiency . 6. Code Quality & Best Practices Maintain high-quality, readable, and maintainable code following coding standards, version control, and documentation best practices . Ensure proper versioning and documentation for all DAGs and workflows. 7. Troubleshooting & Issue Resolution Investigate and resolve Airflow-related issues such as task failures, data inconsistencies, and performance degradation . Ensure high system reliability with automated monitoring and alerting mechanisms . 8. CI/CD & Automation Implement and manage CI/CD pipelines for Airflow DAGs , ensuring efficient and reliable deployment cycles . 9. Data Quality & Security Enforce data integrity, quality, and security through governance policies and compliance measures . 10. Documentation & Knowledge Sharing Create comprehensive documentation for workflows, architecture , and deployment processes . Required Skills & Qualifications: Technical Expertise: 7-8 years of professional experience in data engineering or software development . At least 3-4 years of hands-on experience with Apache Airflow . Deep understanding of Airflow architecture, DAG creation, task scheduling, and monitoring . Experience with Airflow 2.x features such as dynamic pipelines, REST API, and Scheduler improvements . Strong Python programming skills , particularly for data processing and automation . Experience with Python libraries such as Pandas, NumPy, and PySpark . Cloud & Database Skills: Hands-on experience with AWS (S3, EC2, Lambda, RDS, Redshift). Strong SQL skills and experience with relational databases (PostgreSQL, MySQL, etc.). Familiarity with workflow orchestration tools such as Kubernetes, Celery . Development & Collaboration: Experience with Git or other version control systems for managing source code. Strong problem-solving and troubleshooting skills . Ability to mentor junior engineers and collaborate with cross-functional teams . Familiarity with Agile methodologies and tools like Jira, Confluence .