Job Description

Greetings from Clairvoyant an EXL Company We are hiring Lead Data Engineer's at our organization, kindly go through the below job description. Job Description: 8+ years of data management experience. Technically strong in Python, PySpark, Hadoop, Hive Experienced in data integration projects working for large clients using the above tech stacks. Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders. . Replicate and optimize SQL Server stored procedures in Snowflake, ensuring functional accuracy and performance in the cloud environment. Implement and manage automated workflows using Prefect Scheduler to streamline ETL/ELT processes, ensuring reliability, monitoring and alerting for critical workflows. Design and maintain CI/CD pipelines using tools such as Jenkins or GitLab, automating testing, deployment and version control. Collaborate with cross-functional teams, including data scientists, analysts and business stakeholders, to understand data requirements and deliver impactful solutions, simplifying technical concepts for non-technical stakeholders. Use version control systems like GitLab to manage collaborative code development, branching and versioning while fostering best practices in code management. Continuously monitor, troubleshoot and optimize data pipelines for performance, scalability and compliance with SLAs. Develop reconciliation frameworks and implement automated migration validation processes to ensure data consistency and accuracy during cloud migrations. Maintain comprehensive documentation for data pipelines, architectures and processes to enable effective knowledge sharing.