Senior Data Pipeline Engineer

11 hours ago

Sydney, New South Wales, Australia beBeeDataPipelineEngineer Full time $120,000 - $150,000
Senior Data Pipeline Engineer

Role Overview

We are seeking a highly skilled Senior Data Pipeline Engineer to drive the evolution of an existing proof-of-concept (PoC) into a robust and scalable production-ready service.

This individual will be responsible for designing, implementing, and maintaining large-scale data pipelines that support ongoing monitoring and management. They will collaborate closely with cross-functional teams to ensure seamless delivery within tight timelines.

The ideal candidate will possess strong technical expertise in Python, particularly with PySpark, as well as experience working with Databricks and Unity Catalog. Additionally, they should have advanced knowledge of RESTful APIs, CI/CD pipelines, and end-to-end development practices.

Key Responsibilities:

  • Lead the evolution of the PoC into a production-ready service.
  • Design, implement, and maintain large-scale data pipelines.
  • Collaborate with developers and stakeholders to ensure timely delivery.
  • Apply best practices in CI/CD and production deployment.

Required Skills and Qualifications:

  • Python: Strong to advanced proficiency, especially with PySpark.
  • Databricks: Solid experience with Unity Catalog and Delta Live Tables.
  • RESTful APIs: Advanced experience managing client credential requests.
  • CI/CD: Deep understanding of best practices; confident in setting up and maintaining pipelines.
  • End-to-End Development: Experience pushing solutions into production environments.

Nice-to-Have Skills:

  • SQL: Useful for data manipulation and validation.
  • Microsoft Graph API: Great if you've worked with it, but general RESTful API experience is fine.

Why Join Us?

As a Senior Data Pipeline Engineer at our company, you will have the opportunity to work on complex projects, collaborate with talented individuals, and contribute to the growth and success of our organization.

  • Data Engineer

    1 day ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Data Engineer

    2 days ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Data Engineer

    3 days ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Senior Data Engineer

    4 days ago

    Sydney, New South Wales, Australia beBeeData Full time

    About the RoleAs a seasoned Data Engineer, you will design and build cutting-edge data pipelines on Databricks and related cloud platforms. Collaborating closely with data scientists, analysts, and engineers, you will ensure our data infrastructure supports advanced analytics and business insights across various industries.Key Responsibilities:Develop robust...

  • Data Engineer

    4 days ago

    Sydney, New South Wales, Australia Data#3 Full time $90,000 - $120,000 per year

    3 month initial contract + potential extensionMix of WFH and Central Sydney officeOverview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality...

  • Data Pipeline Specialist

    6 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $270,000

    Job Title: Data Engineer AssociateWe are looking for a highly skilled and motivated Data Engineer Associate to join our team. As a key member of our data engineering group, you will play a crucial role in designing, developing, and maintaining our modern data pipelines. You will be working closely with experienced engineers to develop solutions that improve...

  • Senior Data Pipeline Specialist

    6 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $90,000 - $120,000

    We're seeking a skilled Data Engineer to join our high-performing team. As a key member, you'll play a pivotal role in building and maintaining robust data pipelines that drive business growth.Key Responsibilities:Data Pipeline Development: Design, develop, and deploy scalable data pipelines using cutting-edge tools and technologies such as Apache Spark,...

  • Senior Data Pipeline Engineer

    7 days ago

    Sydney, New South Wales, Australia beBeeMachine Full time $120,000 - $140,000

    We are looking for an experienced engineer to join a team focused on scaling and maintaining machine learning platforms.Key Responsibilities:Design, build, and maintain end-to-end data pipelinesCollaborate with data scientists to deploy models into productionImplement automated workflows for model deploymentMonitor and manage model performance metricsDrive...

  • Data Pipeline Engineer

    6 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $180,000

    About the RoleWe are seeking a skilled data engineer to join our team. The role is responsible for managing Drive's data pipelines and creating integrations with platforms, vendors, and third-party data providers.You will be responsible for designing, implementing, and continuously expanding data pipelines by performing extraction, transformation, and...

  • Principal Data Pipeline Specialist

    1 week ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $80,000 - $120,000

    Job OverviewWe're seeking a talented Data Engineer to support the development and maintenance of modern data pipelines. In this role, you'll gain hands-on experience with tools like DBT, Airflow, Snowflake, and Databricks while working alongside experienced engineers to design data models, improve data quality, and document solutions.Data Engineering...