Data Pipeline Engineer

3 days ago

Melbourne, Victoria, Australia beBeeData Full time $140,000 - $170,000
Job Description">

As a Data Engineer, you will play a key role in automating and optimizing data workflows to ensure efficiency and scalability. You will be responsible for designing and implementing CI/CD pipelines, developing infrastructure-as-code templates, and automating ETL/ELT workflow management.

Key Responsibilities:
  • Design and implement CI/CD pipelines for automated data workflow deployment
  • Develop IaC templates for provisioning cloud-based data environments/resources
  • Automate ETL/ELT workflow management using orchestration tools like Apache Airflow or Prefect
  • Optimize data ingestion, transformation, and processing pipelines for scalability and efficiency
Cloud Infrastructure & Security
  • Manage data infrastructure in AWS, Azure, or GCP, ensuring high availability and security
  • Implement RBAC, encryption, and data security best practices
  • Collaborate with Security teams to ensure compliance with industry regulations
  • Monitor cloud costs and optimize resource usage for storage, compute, and networking
Monitoring, Observability & Incident Response
  • Develop real-time monitoring dashboards and alerts for data pipeline performance and failures
  • Use observability tools like Datadog, Prometheus, or Grafana to track latency, errors, and throughput
  • Establish incident response workflows and automated rollback strategies for failures
  • Work with SRE and DevOps teams to maintain high system uptime and resilience

Required Skills and Qualifications:

  • Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field
  • 3+ years of experience in DevOps, Data Engineering, or Cloud Infrastructure roles
  • Experience with CI/CD tools (GitHub Actions, Jenkins, etc.)
  • Proficiency in orchestration tools (Apache Airflow, Prefect, etc.)
  • Strong understanding of ETL/ELT pipelines, workflow scheduling, and automation
  • Hands-on experience with cloud data platforms (AWS, Azure, GCP)
  • Proficiency in Terraform for IaC
  • Experience with observability tools (Datadog, Prometheus, Grafana, etc.)
  • Strong troubleshooting skills for data pipeline failures, latency, and performance optimization
  • Proficiency in Python, Bash, or PowerShell for automation and scripting
  • Experience with SQL and NoSQL databases
  • Ability to work with cross-functional teams
  • Strong documentation skills

  • Data Pipeline Architect

    2 days ago

    Melbourne, Victoria, Australia beBeeData Full time $120,000 - $160,000

    About our organization:We support millions of Australians in comparing and purchasing personal finance and household products.Our company makes Australians' lives easier by saving them time, money, and effort. With over 20 years of experience, we are a one-stop destination for comparison and purchasing across insurance, utilities, and personal finance...

  • Data Pipeline Specialist

    6 days ago

    Melbourne, Victoria, Australia beBeeDataPipeline Full time

    Job Title:Data Pipeline Specialist">About Us:We are a forward-thinking organization that simplifies the process of making informed financial decisions. Our mission is to make a positive impact in the lives of our customers by providing seamless data delivery across the organization.iSelect and Compare the Market: At iSelect and Compare the Market, we're...

  • Data Engineer

    13 hours ago

    Melbourne, Victoria, Australia beBeeDataEngineer Full time $90,000 - $120,000

    About This RoleWe're seeking a skilled professional to drive our data pipeline efficiency and scalability.Job SummaryThis role focuses on optimizing data workflows, implementing cloud infrastructure, and ensuring seamless data delivery across the organization.Key ResponsibilitiesData Pipeline Automation: Design, implement, and maintain CI/CD pipelines for...

  • Data Pipeline Architect

    1 day ago

    Melbourne, Victoria, Australia beBeeDataEngineer Full time $122,736 - $132,847

    Key Data Engineer RoleThe Australian BioCommons team is seeking a highly skilled data engineer to contribute to the establishment of OMIX3, Australia's first integrated, multi-omic analysis platform.Primary Responsibilities:Pipeline Development: Design and implement production-grade tools for automated data flows, including archiving, cataloguing, and...

  • Data Pipeline Architect

    2 days ago

    Melbourne, Victoria, Australia beBeeDataConsultant Full time $150,000 - $180,000

    Senior Data ConsultantAs a seasoned Senior Data Consultant, you will be responsible for developing and delivering high-quality data solutions that drive business value.You will work closely with cross-functional teams to design, build, and maintain scalable and reliable data pipelines that extract, transform, and load (ETL/ELT) data from various sources into...

  • Data Pipeline Specialist

    1 day ago

    Melbourne, Victoria, Australia beBeeETLDeveloper Full time $125,000 - $175,000

    Job Overview">The ETL Developer is responsible for crafting and maintaining robust data pipelines that extract, transform, and load data from diverse sources into centralized repositories.">Key Responsibilities:">">Design, develop, test, and maintain data pipelines to extract, transform, and load data from various sources into centralized...

  • Building Scalable Data Pipelines

    1 day ago

    Melbourne, Victoria, Australia beBeeDataEngineering Full time $190,000 - $220,000

    Job Title: Data EngineerJob Description:We are seeking a skilled Data Engineer to join our team. The successful candidate will be responsible for designing, building, and maintaining scalable data pipelines and ETL processes.This role requires a strong understanding of cloud data architecture and big data platforms. You will work closely with data...

  • Physics Engineer

    16 hours ago

    Melbourne, Victoria, Australia beBeeEngineer Full time $85,000 - $98,000

    Physics Engineer - Pipeline and Engine SystemsThis role involves working with a team to develop, design, and optimize physics pipeline and engine systems. As an experienced engineer, you will communicate features, collaborate with engineering, design, art and productions teams, participate in the design, performance and code review process, and contribute to...

  • Chief Data Pipeline Architect

    13 hours ago

    Melbourne, Victoria, Australia beBeeDataOperationsEngineer Full time $144,000 - $168,000

    As a Senior Data Operations Engineer, you will play a pivotal role in designing and developing data pipelines and integration frameworks for our enterprise data platform. Your expertise will ensure the seamless operation, support, and maintenance of data products and platform services.This is an exciting opportunity to drive improvements in data and...

  • Data Pipelining Specialist

    1 day ago

    Melbourne, Victoria, Australia beBeeDeveloper Full time $100,000 - $140,000

    Job OverviewAs a ETL Developer, you will be responsible for designing and implementing robust data pipelines to extract, transform, and load data from various sources into data warehouses.The ideal candidate will have experience in Python programming and strong analytical skills. They will be able to write clean, efficient, and logical code and provide...