Data Engineer (GCP Bigquery) - Gurugram

Job Description

Data Pipeline Development:- Design & guiding developers in develop and maintain data processing pipelines using PySpark. Collaborate with data engineers and analysts to integrate data from various sources. Implement data transformations, aggregations, and enrichment processes. Performance Optimization Guiding in optimize Spark jobs for performance and scalability. Monitor and troubleshoot performance issues in Spark applications. Implement best practices for efficient Spark code and resource utilization Data Model Design & Development Understanding of Life Insurance Industry leading to design & establish data models in GCP cloud and adhering to enterprise standards. Evaluate data related tools and technologies and recommend appropriate implementation patterns and standards methodologies to ensure latest and modern data architecture & data rules implementation. Create and maintain conceptual / logical data models to identify key business entities and visual relationship resulting in efficient data pipelines and optimized and non-redundant data. Reviewing data models with both technical and business audiences Data Quality and Integrity: ** Ensure data quality and integrity through rigorous testing and validation. Implement data validation and cleansing processes. Design & develop automated data quality checks and monitoring systems