Manager Data Engineer

Job Description

Job Description: We are looking for a highly motivated and skilled L4 Data Engineer Manager, particularly with Databricks experience, is generally responsible for leading data engineering projects, overseeing a team of data engineers, and designing data architectures on platforms like Databricks. Responsibilities: Leadership and Team Management: Lead, mentor, and develop a team of data engineers, ensuring high performance and career growth. Collaborate with stakeholders across data science, analytics, and engineering teams to deliver high-impact data solutions. Oversee the data engineering project lifecycle, from requirements gathering to deployment, ensuring quality and timeliness. Data Architecture and Strategy: Develop scalable data architectures and solutions on Databricks for ETL processes, data warehousing, and big data processing. Define and enforce data governance policies, best practices, and standards for data processing and management. Design data flow pipelines for efficient data ingestion, storage, and processing in cloud environments (e.g., AWS, Azure, GCP). Databricks Management and Optimization: Optimize Databricks clusters for cost and performance efficiency, leveraging cluster scaling and resource management best practices. Implement advanced data transformations and data models within Databricks using Spark and Delta Lake. Ensure integration between Databricks and other data tools, such as data lakes, SQL databases, and BI tools. Data Quality and Security: Monitor and ensure data quality, reliability, and security within the Databricks environment. Implement data validation checks, data profiling, and error-handling mechanisms. Collaborate with security teams to ensure compliance with data privacy regulations and internal security standards. Technical Development and Innovation: Stay updated with the latest Databricks capabilities and cloud technology trends to introduce innovative data engineering practices. Develop reusable and efficient code, libraries, and tools to automate and streamline data workflows. Troubleshoot and resolve complex data pipeline issues and provide continuous improvements in performance and reliability. Skills: Advanced experience with Databricks, PySpark and data pipeline frameworks. Proficiency in Python, SQL, and/or Scala. Familiarity with cloud platforms like AWS, Azure, or GCP. 8+ years in data engineering, with 2+ years in a leadership or managerial role. Strong experience in data architecture, data pipeline optimization, and data modeling. Proven experience in managing large-scale data processing systems and ETL pipelines. Familiarity with Airflow for workflow orchestration and experience with Linux administration and shell scripting. Excellent communication skills to collaborate effectively with cross-functional teams. Strong problem-solving abilities and attention to detail.