Current jobs related to Data Pipeline Architect - Sydney, New South Wales - beBeeDataEngineering

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Full time $120,000 - $160,000

    Job Opportunity: Data ArchitectWe are seeking a highly skilled Data Architect to design and implement cutting-edge data solutions in Azure. This individual will work closely with technical teams to develop efficient, high-quality data pipelines.Key Responsibilities:Design, develop, and maintain scalable data architectures using Databricks and Spark...

  • Senior Data Infrastructure Specialist

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Architect Full time $160,000 - $200,000

    Job SummaryWe are seeking an accomplished data architect to spearhead the development of a hybrid on-premises and cloud-based data platform. This position demands expertise in designing scalable, high-performance data infrastructure that supports trading, research, and operations by ensuring accurate, timely, and accessible data.About the RoleCreate...

  • Data Pipeline Architect

    2 weeks ago

    Sydney, New South Wales, Australia Bebeeintegration Full time

    Job Title: Data Integration SpecialistWe are on a mission to be a force for good, through our people, products and purpose.We seek individuals who are passionate, able to work at pace, comfortable with ambiguity, and committed to making a positive difference.Key Responsibilities:Design, build and optimize pipelines using AWS Glue, dbt, Airflow or similar...

  • Data Pipeline Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeIntegration Full time $150,000 - $200,000

    Job Title: Data Integration SpecialistWe are on a mission to be a force for good, through our people, products and purpose.We seek individuals who are passionate, able to work at pace, comfortable with ambiguity, and committed to making a positive difference.Key Responsibilities:Design, build and optimize pipelines using AWS Glue, dbt, Airflow or similar...

  • Data Pipeline Specialist

    1 week ago

    Sydney, New South Wales, Australia beBeeAutomation Full time $140,000 - $170,000

    Systems ArchitectWe are a global financial trading firm with a drive to deeply understand the world around us and make optimal business decisions every day.Your role involves performing support, deployment, and automation tasks related to data pipelines throughout the Asia Pacific region as part of our trading technology team.This role is integral to our...

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Full time $180,000 - $220,000

    Job OverviewAs a skilled data architect, you will be responsible for designing and implementing scalable data architecture to meet the needs of our organization. Your expertise in data engineering and pipelines will enable you to drive business outcomes through innovative solutions.Key Responsibilities:Develop and maintain robust data pipelines using ETL/ELT...

  • Lead Data Pipeline Developer

    2 weeks ago

    Sydney, New South Wales, Australia Bebeedataengineering Full time

    Software Engineer PositionWe are seeking an experienced software engineer to collaborate with our global teams and contribute to the development of cutting-edge software solutions that support large-scale data processing and analytics.Job Description:We design and build custom software components for data pipelines using Python or other programming...

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    OverviewDrive long-term value and innovation through data-led transformationBe involved in key transformation projectsMake Your Mark – Newly Created PositionAbout usEstablished in 2007, CDC Data Centresis Australia's leading operator of sovereign, secure, world-class data centre facilities.Headquartered in Canberra with multiple facilities in the ACT, NSW,...

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    Drive long-term value and innovation through data-led transformation Be involved in key transformation projects Make Your Mark – Newly Created Position About us:Established in 2007, CDC Data Centres is Australia's leading operator of sovereign, secure, world-class data centre facilities.Headquartered in Canberra with multiple facilities in the ACT, NSW,...

  • Lead Data Pipeline Developer

    2 weeks ago

    Sydney, New South Wales, Australia beBeeDataEngineering Full time $120,000 - $160,000

    Software Engineer PositionWe are seeking an experienced software engineer to collaborate with our global teams and contribute to the development of cutting-edge software solutions that support large-scale data processing and analytics.Job Description:We design and build custom software components for data pipelines using Python or other programming...

Data Pipeline Architect

2 weeks ago

Sydney, New South Wales, Australia beBeeDataEngineering Full time $170,000 - $200,000

Job Description:

  • As a Lead Data Engineer, you will design, build, and optimize data pipeline solutions on Databricks and related cloud platforms. You will work closely with data scientists, analysts, and engineers to ensure our data infrastructure supports advanced analytics and business insights across industries.
  • You will contribute to Data Architecture and Solution Design, helping to build Proof of Concepts. You will also design, develop, and maintain robust ETL/ELT pipelines using Databricks along with AWS/Azure/GCP tools and services to ingest, process, and transform large datasets.
  • You will implement data validation, cleansing, and governance procedures to guarantee data quality, integrity, and security. This includes enforcing data standards and addressing data quality issues proactively.
  • You will continuously improve the scalability, efficiency, and cost-effectiveness of data pipelines. You will identify opportunities to enhance performance, reliability, and cost-efficiency across our data systems.
  • You will monitor data pipeline performance and promptly troubleshoot any issues or failures to ensure high data availability and consistency. You will leverage observability tools and best practices to maintain reliable pipelines.
  • You will develop streaming or event-driven data processes as needed for real-time analytics, leveraging frameworks like Apache Kafka and Spark Structured Streaming.
  • You will maintain clear documentation of data pipelines, data models, and processes for transparency and team knowledge sharing. You will follow best practices in coding, testing, and version control to ensure maintainable and auditable workflows.

Required Skills and Qualifications:

  • Proficiency in Python for data engineering (including PySpark and libraries like pandas/Polars) and in SQL for data querying and transformation.
  • Solid understanding of data warehousing concepts and dimensional data modeling (e.g. star schema, Kimball methodology).
  • Hands-on experience with relational database systems and SQL (e.g. SQL Server, PostgreSQL) and familiarity with NoSQL databases (e.g. MongoDB, Cassandra) for varied data storage needs.
  • Strong experience designing and implementing ETL/ELT processes and integrating data from multiple sources.
  • Proven experience working with multiple cloud data platforms such as AWS/Azure/GCP.
  • Expertise in Databricks and the Spark ecosystem for large-scale data processing is required.
  • Familiarity with data pipeline orchestration and automation tools and with CI/CD pipelines for deploying data workflows.
  • Experience monitoring data pipeline performance and using observability tools to ensure data reliability is a plus.
  • Experience with designing and building event-driven architectures and streaming data tools (such as Apache Kafka or Spark Streaming) is beneficial for handling real-time data flows.
  • Experience working in Agile teams with iterative development, and a collaborative approach to problem-solving.
  • Holding a current Databricks certification (e.g. Databricks Certified Data Engineer) is a strong advantage.
  • Background in or understanding of data from the energy, resources, or mining industry is a plus, as it will help in delivering business-focused insights in these sectors.

Benefits:

  • Competitive salary package
  • Share plan
  • Company performance bonuses
  • Value-based recognition awards
  • Referral bonus

Others:

  • Hybrid work and flexible working hours
  • Employee assistance program
  • Global internal wellbeing program
  • Access to wellbeing apps
  • Monthly Lifestyle Allowance
  • Novated Leasing
  • Loyalty Leave
  • Inclusive Parental Leave Policy
  • Work From Anywhere