Lead Data Engineer

Job Description

We are seeking a highly skilled Lead Data Engineer to join our data engineering team for an on-premise environment. A large portion of your time will be in the weeds working alongside your team architecture, designing, implementing, and optimizing data solutions. The ideal candidate will have extensive experience in building and optimizing data pipelines, architectures, and data sets, with a strong focus on Python, SQL, Hadoop, HDFS, and Apache NiFi. What you ll be doing? Design, develop, and maintain robust, scalable, and high-performance data pipelines and data integration solutions. Manage and optimize data storage in Hadoop Distributed File System (HDFS). Design and implement data workflows using Apache NiFi for data ingestion, transformation, and distribution. Collaborate with cross-functional teams to understand data requirements and deliver efficient solutions. Ensure data quality, governance, and security standards are met within the on-premise infrastructure. Monitor and troubleshoot data pipelines to ensure optimal performance and reliability. Automate data workflows and processes to enhance system efficiency. Bachelor s degree in computer science, Software Engineering, or a related field 6+ years of experience in data engineering or a related field Strong programming skills in Python and SQL. Hands-on experience with Hadoop ecosystem (HDFS, Hive, etc.). Proficiency in Apache NiFi for data ingestion and flow orchestration. Experience in data modeling, ETL development, and data warehousing concepts. Strong problem-solving skills and ability to work independently in a fast-paced environment. Good understanding of data governance, data security, and best practices in on-premise environments. Good to Have: Experience with other big data tools like Spark, Kafka, etc.