Job Description

Responsibilities : Data Pipeline Development:Design, develop, and maintain scalable and reliable data pipelines using Python, Spark, and other relevant technologies. Extract, transform, and load (ETL) data from various sources into data lakes and data warehouses. Implement data quality checks and monitoring to ensure data accuracy and consistency. Big Data Technologies:Work with Hadoop, Spark, Confluent Kafka, and other big data technologies to process and analyze large datasets. Build and maintain data lakes for storing structured and unstructured data. Optimize data processing workflows for performance and efficiency. Cloud Platform Integration:Develop and deploy data solutions on cloud platforms such as Azure or Google Cloud. Utilize cloud-based data services like Azure Data Factory, Redshift, Snowflake, or BigQuery. Implement cloud-based data storage and processing solutions. Database Management:Design and implement database schemas in PostgreSQL and other SQL/NoSQL databases. Write complex SQL queries for data extraction and analysis. Optimize database performance and ensure data integrity. Data Streaming:Implement data streaming solutions using Confluent Kafka. Build real-time data pipelines for data ingestion and processing. Data Warehousing:Design and implement data warehousing solutions. Work with data models and dimensional modeling. Key skills Hadoop, Spark, Confluent, Kafka, Data Lake, PosgreSQL, Data Factory etc. Python, Scala, or Java, Confluent, Azure, Google Cloud, SQL, NoSQL databases, Redshift, Snowflake, BigQuery