Job Description

Job Summary:As a Senior Data Engineer, you will be responsible for designing, implementing, and optimizing scalable data systems. Your expertise in building Data Lakes and working with modern data processing tools such as Databricks will be pivotal in transforming large data sets into actionable insights. You will be expected to work closely with cross-functional teams, including Data Scientists, Data Analysts, and Cloud Engineers, to deliver solutions that enable advanced analytics and machine learning models in the cloud.Key Responsibilities:Data Lake Architecture: Design and implement robust, scalable Data Lakes using Databricks and cloud technologies (AWS, Azure, GCP) to manage vast amounts of structured and unstructured data.Data Engineering: Build and optimize ETL pipelines to ingest, transform, and clean data from various sources for analytics and machine learning.AI/ML Integration: Collaborate with data scientists to implement AI/ML models and ensure seamless integration with production data pipelines.Cloud Expertise: Leverage cloud platforms (AWS, Azure, GCP) for data storage, compute, and orchestration to ensure high performance and cost efficiency.Forecasting Models: Develop and deploy forecasting models using historical data, ensuring accuracy and scalability in production environments.Performance Tuning: Optimize data pipelines and queries for performance, cost-efficiency, and scalability in a cloud-native environment.Collaboration: Work closely with product, analytics, and business teams to ensure data solutions align with business objectives and customer needs.Mentorship: Guide and mentor junior team members, sharing best practices and improving the overall data engineering capabilities of the team.Documentation: Ensure that data models, pipelines, and architectures are well-documented for long-term maintainability.Required Qualifications:Experience: 5-8 years of hands-on experience in Data Engineering, with a proven track record of designing and deploying Data Lakes in a cloud-based environment.Data Lake Expertise: Strong expertise in building Data Lakes from scratch using Databricks, and other big data technologies (e.g., Spark, Hadoop, Delta Lake).Cloud Platforms: Proficiency in working with cloud platforms such as AWS, Azure, or GCP, especially in the context of data engineering and processing.Data Pipelines: Advanced skills in building and optimizing scalable ETL/ELT pipelines, data integration, and automation.Machine Learning/AI: Practical experience with implementing AI/ML solutions on live projects, including deploying models into production environments.Forecasting: Experience in building and maintaining forecasting models (e.g., time series forecasting, predictive analytics) and integrating them into data pipelines.Programming: Strong coding skills in Python, SQL, and experience with Spark/Scala.Big Data Frameworks: Experience with distributed computing frameworks such as Apache Spark and Databricks.Version Control: Proficiency in Git, CI/CD, and version control practices.Problem-Solving: Strong analytical and problem-solving skills with the ability to troubleshoot complex issues across the data pipeline.Communication: Excellent written and verbal communication skills, with the ability to translate complex technical concepts to non-technical stakeholders.Preferred Qualifications:AI/ML Deployment: Hands-on experience in deploying AI/ML models in production environments, preferably using Databricks and cloud services.Data Governance: Knowledge of data governance, security, and privacy best practices in a cloud environment.Data Visualization Tools: Familiarity with tools like Tableau, Power BI, or Looker to assist in visualizing data pipelines and analytics results.Certifications: Cloud certifications (AWS Certified Solutions Architect, Microsoft Certified: Azure Data Engineer, Google Cloud Professional Data Engineer) or Databricks certifications are a plus.