Data Pipeline Engineer

1 day ago

Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $180,000
About the Role

We are seeking a skilled data engineer to join our team. The role is responsible for managing Drive's data pipelines and creating integrations with platforms, vendors, and third-party data providers.

You will be responsible for designing, implementing, and continuously expanding data pipelines by performing extraction, transformation, and loading activities.

This is a hands-on role where you will understand business needs to create efficient data flows within technical constraints, choosing the right tools to define how data should move, transform, and land within our systems.

Key Responsibilities:
  • Design and implement data pipelines to extract, transform, and load data from various sources.
  • Work with the business to design and deliver correct, high-quality data to meet functional requirements.
  • Maintain and improve existing processes.
  • Create detailed validation checks to identify schema drift, data duplication, and integrity errors.
  • Maintain comprehensive documentation of pipeline logic and data lineage.
  • Implement data governance policies by masking or excluding sensitive information and ensuring that all processing logs support regulatory traceability requirements.
  • Fine-tune SQL queries to reduce data load times and monitor overall system performance.
  • Build robust error handling mechanisms to minimize downtime and ensure data integrity during recovery processes.
  • Investigate logs and data structures to identify root causes and resolve issues quickly.
  • Implement CDC (Change Data Capture) through log-based tracking, query-based monitoring, or API-driven change detection.

To succeed in this role, you will bring:

  • A Bachelor's degree in computer science, information technology, or a related field.
  • 5+ years of experience with ETL processes, data warehousing, and data modeling.
  • Proven track record of designing, building, and maintaining robust data pipelines to integrate data from diverse sources.
  • Strong proficiency in SQL and Python for data manipulation, automation, and pipeline development.
  • Hands-on experience working within a major cloud environment such as AWS is a must.

Nice-to-Haves:

  • Demonstrated experience in evaluating, selecting, and implementing new ETL tools, with familiarity in modern platforms like CDATA, Airflow, or Airbyte.
  • Database and warehouse proficiency across systems like PostgreSQL, Snowflake, BigQuery, and Redshift.
  • Deep understanding of data consolidation strategies to support standardized reports and data exploration capabilities for business stakeholders.

We welcome discussing your flexibility needs during the hiring process.

We encourage applications from people with disabilities, and of all ages, genders, nationalities, backgrounds and cultures as we recognise the importance and value of diverse perspectives.

  • Senior Data Engineer

    4 hours ago

    Sydney, New South Wales, Australia beBeeData Full time

    About the RoleAs a seasoned Data Engineer, you will design and build cutting-edge data pipelines on Databricks and related cloud platforms. Collaborating closely with data scientists, analysts, and engineers, you will ensure our data infrastructure supports advanced analytics and business insights across various industries.Key Responsibilities:Develop robust...

  • Data Pipeline Specialist

    1 day ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $270,000

    Job Title: Data Engineer AssociateWe are looking for a highly skilled and motivated Data Engineer Associate to join our team. As a key member of our data engineering group, you will play a crucial role in designing, developing, and maintaining our modern data pipelines. You will be working closely with experienced engineers to develop solutions that improve...

  • Senior Data Pipeline Engineer

    3 days ago

    Sydney, New South Wales, Australia beBeeMachine Full time $120,000 - $140,000

    We are looking for an experienced engineer to join a team focused on scaling and maintaining machine learning platforms.Key Responsibilities:Design, build, and maintain end-to-end data pipelinesCollaborate with data scientists to deploy models into productionImplement automated workflows for model deploymentMonitor and manage model performance metricsDrive...

  • Data Pipeline Specialist

    3 days ago

    Sydney, New South Wales, Australia beBeeDevopser Full time $150,000 - $170,000

    Job DescriptionAs a DevOps Engineer, you will play a vital role in our trading technology team. You will be responsible for performing daily monitoring and market pre-open system checks utilizing automation, developing new tools and utilities to help build data pipelines and studies, and becoming a subject matter expert in our in-house systems and data...

  • Data Pipeline Specialist

    1 week ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time

    About the RoleDrive's Data Operations team is seeking an ETL Engineer to help us achieve our vision of shaping the future of automotive in Australia. As a key member of this team, you will be responsible for designing, implementing, and maintaining efficient data pipelines that drive business value.The ideal candidate will have 5+ years of experience with...

  • Data Pipeline Architect

    6 hours ago

    Sydney, New South Wales, Australia beBeeDataEngineering Full time $170,000 - $200,000

    Job Description:As a Lead Data Engineer, you will design, build, and optimize data pipeline solutions on Databricks and related cloud platforms. You will work closely with data scientists, analysts, and engineers to ensure our data infrastructure supports advanced analytics and business insights across industries.You will contribute to Data Architecture and...

  • Data Pipeline Specialist

    2 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $100,000 - $140,000

    Data Engineering RoleWe are seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data pipelines to support data ingestion, transformation, and integration from various sources.The ideal candidate will have experience with Kafka, strong knowledge of AWS...

  • Senior Data Pipeline Specialist

    1 day ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $90,000 - $120,000

    We're seeking a skilled Data Engineer to join our high-performing team. As a key member, you'll play a pivotal role in building and maintaining robust data pipelines that drive business growth.Key Responsibilities:Data Pipeline Development: Design, develop, and deploy scalable data pipelines using cutting-edge tools and technologies such as Apache Spark,...

  • Principal Data Pipeline Specialist

    5 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $80,000 - $120,000

    Job OverviewWe're seeking a talented Data Engineer to support the development and maintenance of modern data pipelines. In this role, you'll gain hands-on experience with tools like DBT, Airflow, Snowflake, and Databricks while working alongside experienced engineers to design data models, improve data quality, and document solutions.Data Engineering...

  • Chief Data Pipeline Architect

    4 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $150,000 - $180,000

    Job DescriptionWe are seeking a skilled Data Engineer to develop and maintain robust data pipelines and infrastructure for analytics, machine learning, and reporting platforms.About the RoleThis position involves owning the development and maintenance of data pipelines and data infrastructure to support business needs. The successful candidate will use SQL...