Senior Data Engineer

Job Description

Qualification 3-5 Years of overall IT experience as a Data Engineer with hands on expertise in building data pipeline, data transformation and metadata creation involving large complex and disconnected data sets.Graduate in Computer Science, Statistics, Informatics, Information Systems or another quantitative fieldData Engineering Certification from GCP. Job Description Assemble large, complex data sets that meet functional / non-functional business requirements.Build processes supporting data transformation, data structures, metadata, dependency and workload management.Hands on in manipulating, processing and extracting value from large disconnected datasets.Experience in building and optimizing ‘big data’ data pipelines, architectures and data sets.Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.Experience supporting and working with cross-functional teams in a dynamic environment. Skills/Tools/Techniques Hands on proficiency with big data tools: Hadoop, Hive, GCP BigQuery, GCP DataProc, Python/ScalaNice to have experience in GCP cloud services: Google cloud storage, Big query, Spanner, Cloud Pub/Sub & ComposerExperience in CI/CD using Jenkins/GitlabNice to have experience with stream-processing systems: Dataflow etc.Experience with data pipeline and workflow management tools: Airflow, etc.Strong analytic skills related to working with unstructured datasets.