Job Description

JD: 1. Data Pipeline Development & Orchestration -Design, develop, and orchestrate data pipelines using Apache Airflow (Cloud Composer) to automate data ingestion, transformation, and loading (ETL/ELT) workflows. 2. Data Transformation with DBT - Implement DBT (Data Build Tool) models for transforming raw data into analytics-ready datasets in BigQuery, applying SQL-based transformations, modular modeling, and version control. 3. Stream & Batch Processing with Dataflow - Build real-time (Streaming) and batch data processing pipelines using Apache Beam on Dataflow, ensuring scalable and efficient data processing. 4. BigQuery Optimization & Performance Tuning - Design optimized BigQuery schemas, implement partitioning, clustering, and tune queries for cost-efficiency and performance. 5. Monitoring, Logging, and CI/CD for Data Pipelines - Set up monitoring with Stackdriver, logging, error handling, and CI/CD pipelines for automated testing and deployment of DBT, Airflow DAGs, and Dataflow jobs.