Pyspark Developer

Job Description

Role: Pyspark Developer Location: Hyderabad/ Chennai Duration: Full time Job Description: Key Skills – Pyspark, Spark, Python, Hive, SQL Job Descriptions: - 6+ Year experience as Data Engineer with experience in building ETL/Data pipelines with cloud technologies - 4+ years extensive experience in designing, developing, and implementing scalable data pipelines using Databricks to ingest, transform, and store structured and unstructured data. - 4+ years' experience programming languages such as Python/Pyspark and query languages like SQL - 4+ years' experience in building metadata driven data ingestion pipelines using ADF - 4+ years' experience in analysing, optimizing, and tuning existing data pipelines for performance, reliability, and efficiency - 2+ years' experience in implementing ML Ops practices to streamline the deployment and management of machine learning models. - 2+ years' experience in utilizing Apache Airflow for job orchestration and workflow management - Familiarity with CI/CD tools and practices for automating the deployment of data engineering solutions. - Experience in collaborating with data scientists, analysts, and other stakeholders to understand business requirements and translate them into technical solutions - Knowledge/Experience in implementing security measures and standard processes to ensure data privacy and compliance with regulatory standards - In-depth knowledge of data engineering concepts, ETL processes, and data architecture principles. Thanks, Siva Show more Show less