Data Engineer

Toronto, ON, Canada
11 days ago

Description

We are seeking a skilled Data Engineer to join our growing data team. The ideal candidate will have hands-on experience in building and optimizing large-scale data pipelines using Databricks and PySpark, and orchestrating workflows with Apache Airflow. You'll play a critical role in transforming raw data into actionable insights by developing scalable and reliable data solutions.

Requirements

Qualifications:
  • Proven experience with Databricks and PySpark in a production environment.
  • Strong expertise in Apache Airflow for data orchestration.
  • Proficiency in SQL and Python for data manipulation and automation.
  • Experience working with cloud platforms (e.g., AWS, Azure, or GCP) is a plus.
  • Solid understanding of distributed systems and big data technologies.
  • Experience working in an Agile development environment is preferred.
Preferred:
  • Databricks certifications (e.g., Databricks Certified Data Engineer Associate or Professional).
  • Experience with Delta Lake, MLflow, or similar technologies within the Databricks ecosystem.
  • Familiarity with CI/CD practices and tools (e.g., Git, Jenkins, etc.).

Responsibilities

  • Design, develop, and optimize large-scale data pipelines using Databricks and PySpark for efficient data processing and analytics.
  • Orchestrate and monitor data workflows using Apache Airflow to ensure reliability, efficiency, and scalability.
  • Collaborate with data analysts, data scientists, and business stakeholders to understand data needs and deliver high-quality data solutions.
  • Implement best practices for data engineering including version control, testing, monitoring, and alerting.
  • Ensure data quality, consistency, and governance across all pipelines and datasets.
  • Participate in code reviews, design sessions, and continuous improvement initiatives within the data engineering team.

Additional Information

It's a long term contract role with office-remote hybrid work style from Toronto.

About EffiQ

EffiQ's mission is to empower and elevate businesses with innovative solutions to enhance their digital journeys by modernizing, integrating, and automating. We specialize in delivering cutting-edge solutions that drive efficiency, improve productivity and unlock new levels of operational excellence. EffiQ Technology Solutions is a team of experienced professionals and technology experts who can support and manage enterprise software systems such as Manhattan WMS, SAP and Microsoft D365 ERP.  EffiQ's vision is to revolutionize supply chain.

We also assist organizations to quickly fill skill gaps, manage increased workloads, and maintain productivity without the long-term commitment of permanent hires. 

https://www.effiqt.com/
Apply Now