Leading Data Pipeline Architect

1 week ago

Sydney, New South Wales, Australia beBeeDataEngineer Full time $100,000 - $200,000
Job Description

Achieve success in your career as a Data Engineer by joining our team to design scalable data pipelines and build modern, high-performance data platforms.

Our ideal candidate will have experience with Databricks, Delta Lake, and AWS services.

  • Develop/improve data pipelines using Python and PySpark for batch and streaming workloads
  • Design and manage Delta Lake layers (Bronze, Silver, Gold) - Databricks
  • Set up and maintain Databricks workflows, jobs, notebooks, and DLT pipelines
  • Optimize performance using Databricks tools and best practices for clean, modular code
  • Work with AWS S3 to manage raw and processed data securely
  • Use Terraform and Databricks Asset Bundles
  • Automate AWS workflows using Boto3 (S3, Glue, Lambda)
  • Collaborate with data scientists and engineers to ensure seamless data flow across teams
  • Implement data quality checks, validation, and monitoring

Required Skills & Experience:

  • Strong coding skills in Python and PySpark
  • Experience with Databricks and Delta Lake
  • Proficiency in AWS services, especially S3, Glue, Lambda, and Redshift
  • Knowledge of Terraform or CloudFormation
  • Ability to automate AWS tasks with Boto3
  • Prior experience building efficient, secure, and scalable pipelines
  • Excellent teamwork and communication skills
  • Databricks certification
  • Experience working in fast-paced delivery environments
Key Responsibilities

  • Lead Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Full time $180,000 - $250,000

    Job Title:Lead Data ArchitectJob DescriptionWe are seeking a skilled and experienced Lead Data Architect to join our team. As a key member of our organization, you will be responsible for designing and developing cutting-edge data solutions that drive business growth and innovation.The ideal candidate will have a strong background in data engineering, cloud...

  • Lead Data Pipeline Developer

    1 week ago

    Sydney, New South Wales, Australia Bebeedataengineering Full time

    Software Engineer PositionWe are seeking an experienced software engineer to collaborate with our global teams and contribute to the development of cutting-edge software solutions that support large-scale data processing and analytics.Job Description:We design and build custom software components for data pipelines using Python or other programming...

  • Lead Data Pipeline Developer

    1 week ago

    Sydney, New South Wales, Australia beBeeDataEngineering Full time $120,000 - $160,000

    Software Engineer PositionWe are seeking an experienced software engineer to collaborate with our global teams and contribute to the development of cutting-edge software solutions that support large-scale data processing and analytics.Job Description:We design and build custom software components for data pipelines using Python or other programming...

  • Data Pipeline Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeDataEngineering Full time $170,000 - $200,000

    Job Description:As a Lead Data Engineer, you will design, build, and optimize data pipeline solutions on Databricks and related cloud platforms. You will work closely with data scientists, analysts, and engineers to ensure our data infrastructure supports advanced analytics and business insights across industries.You will contribute to Data Architecture and...

  • Data Architect

    7 days ago

    Sydney, New South Wales, Australia beBeeData Full time $180,000 - $220,000

    Job OverviewAs a skilled data architect, you will be responsible for designing and implementing scalable data architecture to meet the needs of our organization. Your expertise in data engineering and pipelines will enable you to drive business outcomes through innovative solutions.Key Responsibilities:Develop and maintain robust data pipelines using ETL/ELT...

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    OverviewDrive long-term value and innovation through data-led transformationBe involved in key transformation projectsMake Your Mark – Newly Created PositionAbout usEstablished in 2007, CDC Data Centresis Australia's leading operator of sovereign, secure, world-class data centre facilities.Headquartered in Canberra with multiple facilities in the ACT, NSW,...

  • Data Architect

    1 week ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    Drive long-term value and innovation through data-led transformation Be involved in key transformation projects Make Your Mark – Newly Created Position About us:Established in 2007, CDC Data Centres is Australia's leading operator of sovereign, secure, world-class data centre facilities.Headquartered in Canberra with multiple facilities in the ACT, NSW,...

  • Data Pipeline Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Full time $100,000 - $170,000

    Job DescriptionAs a Senior Data Engineer, you will play a pivotal role in crafting and implementing cutting-edge data pipelines that transform raw market data into actionable insights for traders worldwide.Our ideal candidate will possess exceptional technical skills, paired with outstanding collaboration and problem-solving abilities.Key...

  • Data Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeData Full time $120,000 - $160,000

    Job Opportunity: Data ArchitectWe are seeking a highly skilled Data Architect to design and implement cutting-edge data solutions in Azure. This individual will work closely with technical teams to develop efficient, high-quality data pipelines.Key Responsibilities:Design, develop, and maintain scalable data architectures using Databricks and Spark...

  • Data Architect

    1 week ago

    Sydney, New South Wales, Australia Bebeedata Full time

    Job Overview:We're seeking a seasoned data solutions architect to design and implement efficient data access frameworks, ensuring seamless operations of our data pipelines.This role will collaborate closely with researchers and cross-functional teams to drive forward our data strategy, enhancing and aligning our data capabilities.As a key player in our...