Job Description

We are seeking an experienced AWS Data Engineer with expertise in Databricks to design, build, and optimize large-scale data pipelines and data processing workflows on the AWS Cloud. The ideal candidate will have hands-on experience working with Databricks, big data technologies, and AWSnative services, ensuring efficient data ingestion, transformation, and analytics to support businesscritical decisions. Key Responsibilities: Data Pipeline Development: Design, implement, and manage scalable ETL/ELT pipelines using AWS services and Databricks. Data Integration: Ingest and process structured, semi-structured, and unstructured data from multiple sources into AWS Data Lake or Databricks. Data Transformation: Develop advanced data processing workflows using PySpark, Databricks SQL, or Scala to enable analytics and reporting. Databricks Management: Configure and optimize Databricks clusters, notebooks, and jobs for performance and cost efficiency. AWS Architecture: Design and implement solutions leveraging AWS-native services like S3, Glue, Redshift, EMR, Lambda, Kinesis, and Athena. Collaboration: Work closely with Data Analysts, Data Scientists, and other Engineers to understand business requirements and deliver data-driven solutions. Performance Tuning: Optimize data pipelines, storage, and queries for performance, scalability, and reliability. Monitoring and Security: Ensure data pipelines are secure, robust, and monitored using CloudWatch, Datadog, or equivalent tools. Documentation: Maintain clear and concise documentation for data pipelines, workflows, and architecture. Required Skills & Qualifications: Data Engineering Expertise: 6+ years of experience in data engineering with at least 2+ years working on Databricks. AWS Cloud Services: Hands-on experience with AWS ecosystem, including S3, Glue, Redshift, DynamoDB, Lambda, and other AWS data services. Programming Languages: Proficiency in Python (PySpark), Scala, or SQL for data processing and transformation. Databricks: Extensive experience with Databricks Workspace, Delta Lake, and managing databricks jobs and pipelines. Big Data Frameworks: Strong knowledge of Apache Spark for distributed data processing. Data Warehousing: Experience with modern data warehouse solutions, including Redshift, Snowflake, or Databricks SQL. Version Control & CI/CD: Familiarity with Git, Terraform, and CI/CD pipelines for deploying data solutions. Monitoring & Debugging: Experience with tools like CloudWatch, Datadog, or equivalent for pipeline monitoring and troubleshooting. Preferred Qualifications: Certification in AWS Data Analytics or Databricks. Experience with real-time data streaming tools like Kafka, Kinesis, or AWS MSK. Knowledge of data governance and data security best practices. Exposure to machine learning workflows and integration with Databrick