Data Engineer

Job Description

Job Location: Remote Candidate having 4+ years of total experience in Data engineering (Core development/design), in which 3+ years on AWS with strong hands on (AWS glue , pyspark , SQL , Athena , lambda , SNS , S3 )and 1+ year on Databricks. Skill required : Manage the ingestion and processing of event stream data from multiple source systems, Create backend systems that scale by leveraging AWS services and Databricks. Must have Implemented and configured the Databricks environment, including clusters, notebooks, and libraries, ensuring optimal performance and resource utilization. Must be strong in writing unit test case and integration test and Responsible to set best practices around Databricks CI/CD. Must have Integrated data from various sources, including structured, semi-structured, and unstructured data, into Databricks for processing and analysis. Must have knowledge of code and Aws/Databricks cluster optimization and its integration with various cloud services, various governance and security standards, strong performance optimization skills to improve efficiency and reduce cost.