About the RoleWe are seeking a Senior Python/Data Engineer to design, develop, and optimize large-scale data pipelines, transformation workflows, and analytics-ready datasets. This role requires expertise in Python, Apache Airflow, Apache Spark, SQL, and DuckDB, along with strong experience in data quality, data processing, and automation.As a Senior Data Engineer, you will play a key role in building scalable, high-performance data engineering solutions, ensuring data integrity, and supporting real-time and batch data workflows. You will work closely with Data Scientists, Analysts, DevOps, and Engineering teams to build efficient, cost-effective, and reliable data architectures.Key ResponsibilitiesDesign, build, and maintain scalable ETL/ELT data pipelines using Apache Airflow, Spark, and SQL.Develop Python-based data engineering solutions to automate data ingestion, transformation, and validation.Implement data transformation and quality checks for structured and unstructured datasets.Work with DuckDB and other in-memory databases to enable fast exploratory data analysis (EDA).Optimize data storage and retrieval using Parquet, Apache Iceberg, and S3-based data lakes.Develop SQL-based analytics workflows and optimize performance for querying large datasets.Implement data lineage, governance, and metadata management for enterprise-scale data solutions.Ensure high availability, fault tolerance, and security of data pipelines.Collaborate with Data Science, AI/ML, and Business Intelligence teams to enable real-time and batch analytics.Work with cloud platforms (AWS, Azure, GCP) for data pipeline deployment and scaling.Write clean, efficient, and maintainable code following best software engineering practices.Required Skills & Qualifications7+ years of experience in data engineering, big data processing, and backend development.Expertise in Python for data processing and automation.Strong experience with Apache Airflow for workflow orchestration.Hands-on experience with Apache Spark for big data transformations.Proficiency in SQL (PostgreSQL, DuckDB, Snowflake, etc.) for analytics and ETL workflows.Experience with data transformation, data validation, and quality assurance frameworks.Hands-on experience with DuckDB, Apache Arrow, or Vaex for in-memory data processing.Knowledge of data lake architectures (S3, Parquet, Iceberg) and cloud data storage.Familiarity with distributed computing, parallel processing, and optimized query execution.Experience working in CI/CD, DevOps, containerization (Docker, Kubernetes), and cloud environments.Strong problem-solving and debugging skills.Excellent written and verbal communication skills.Preferred Skills (Nice to Have)Experience programming in JAVA/JEE platform is highly desired.Experience with data streaming technologies (Kafka, Flink, Kinesis).Familiarity with NoSQL databases (MongoDB, DynamoDB).Exposure to AI/ML data pipelines and feature engineering.Knowledge of data security, compliance (SOC2 Type2, GDPR, HIPAA), and governance best practices.Experience in building metadata-driven data pipelines for self-service analytics.

Seeking a Python Developer (3-7 years exp) with strong Python, Pandas, and SQL skills. Exposure to ETL, Airflow, and cloud platforms preferred. Apply now!

6+ yrs in Angular (8+), HTML5, CSS (SASS), Bootstrap, JavaScript. Strong UI & front-end dev. 4+ yrs in GoJS (drag & drop tools). Exp. with REST/GraphQL APIs, Git, Scrum. JavaScript unit testing (Jasmine/Mocha/Karma). User Access Control in Angular.