Lead Data Engineer

1 week ago

Brisbane, Queensland, Australia beBeeDataEngineering Full time $120,000 - $180,000
Job Description

We are seeking an accomplished Lead Data Engineer to spearhead the design, development, and optimization of data pipeline solutions on Databricks and related cloud platforms. This role will entail collaborating with cross-functional teams, including data scientists, analysts, and engineers, to deliver advanced analytics and business insights across various industries.


Key Responsibilities:
  • Collaborate with stakeholders to understand data requirements and develop effective pipeline solutions.
  • Contribute to data architecture and solution design, helping to build proof of concepts.
  • Design, develop, and maintain robust ETL/ELT pipelines on Databricks, utilizing AWS/Azure/GCP tools and services to ingest, process, and transform large datasets.
  • Implement data validation, cleansing, and governance procedures to ensure data quality, integrity, and security.
  • Continuously improve the scalability, efficiency, and cost-effectiveness of data pipelines, identifying opportunities to enhance performance, reliability, and cost-efficiency across our data systems.
  • Monitor data pipeline performance and promptly troubleshoot any issues or failures to ensure high data availability and consistency.

Requirements:
  • Proficiency in Python for data engineering, including PySpark and libraries like pandas/Polars, and SQL for data querying and transformation.
  • Solid understanding of data warehousing concepts and dimensional data modeling.
  • Hands-on experience with relational database systems and SQL, as well as familiarity with NoSQL databases for varied data storage needs.
  • Strong experience designing and implementing ETL/ELT processes and integrating data from multiple sources.
  • Proven experience working with multiple cloud data platforms, including AWS/Azure/GCP.
  • Expertise in Databricks and the Spark ecosystem for large-scale data processing.
  • Familiarity with data pipeline orchestration and automation tools, as well as CI/CD pipelines for deploying data workflows.

Benefits:
  • Competitive salary package and share plan.
  • Career coaching and global career opportunities.
  • Complex projects, rotations, and internal tech communities.
  • Hybrid work and flexible working hours.
  • Global wellbeing programme and access to wellbeing apps.

Additional Benefits:
  • Monthly lifestyle allowance for health and wellbeing activities.
  • Novated leasing for pre-tax car leasing benefit.
  • Loyalty leave policy with additional days off.
  • Inclusive parental leave policy.
  • Work-from-anywhere policy.

We value diversity, inclusivity, and respect, and welcome applications from people of all backgrounds, experiences, and perspectives. Hiring decisions are based on merit, skills, qualifications, and potential.

  • Data Engineering Lead

    3 days ago

    Brisbane, Queensland, Australia P&C Partners Pty Ltd Full time

    About the RoleThis is a newly created strategic leadership position within a growing SaaS and digital services organisation. The Data Engineering Lead will play a pivotal role in modernising the company's data infrastructure, leading a small but high-performing team, and delivering scalable, secure, and insightful data solutions across both internal...

  • Data Engineering Lead

    3 days ago

    Brisbane, Queensland, Australia P&C Partners Pty Ltd Full time

    About the RoleThis is a newly created strategic leadership position within a growing SaaS and digital services organisation. The Data Engineering Lead will play a pivotal role in modernising the company's data infrastructure, leading a small but high-performing team, and delivering scalable, secure, and insightful data solutions across both internal...

  • Data Engineer Lead

    2 weeks ago

    Brisbane, Queensland, Australia beBeeDataEngineeringManager Full time $180,000 - $250,000

    Job Title: Data Engineering Manager","The role of Data Engineering Manager involves leading and growing a global team of data services, overseeing the deployment, maintenance, and scaling of cloud-native platforms. This position is crucial in ensuring the scalability, reliability, and performance of data infrastructure.","About The Role:","Lead and mentor a...

  • Lead Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia CYOS Solutions Full time

    Join to apply for the Lead Data Engineer role at CYOS Solutions1 day ago Be among the first 25 applicantsJoin to apply for the Lead Data Engineer role at CYOS SolutionsGet AI-powered advice on this job and more exclusive features.Direct message the job poster from CYOS SolutionsTalent Acquisition Specialist @ CYOS Solutions | IT RecruitmentApplication...

  • Lead Data Engineer

    3 days ago

    Brisbane, Queensland, Australia CYOS Solutions Full time

    Join to apply for the Lead Data Engineer role at CYOS Solutions1 day ago Be among the first 25 applicantsJoin to apply for the Lead Data Engineer role at CYOS SolutionsGet AI-powered advice on this job and more exclusive features.Direct message the job poster from CYOS SolutionsTalent Acquisition Specialist @ CYOS Solutions | IT RecruitmentApplication...

  • Lead Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Integrant, Inc. Full time

    We're on the lookout for exceptional individuals to join our team asLead Data Engineers.As a part of our team, you'll be driving change and making a significant impact in the gaming industry.Lead the design, development, and maintenance of end-to-end data analytical solutions using Microsoft on-premises and cloud technologies.Engage in collaborative...

  • Strategic Data Engineering Lead

    3 days ago

    Brisbane, Queensland, Australia beBeeDataEngineering Full time $140,000 - $180,000

    Job SummaryWe are seeking a highly skilled Strategic Data Engineering Lead to lead the design, development, and optimization of our enterprise data platform. This is an exciting opportunity to shape the strategy for scalability, reliability, and performance across the data ecosystem.About the RoleThe successful candidate will have a proven track record in...

  • Data Engineering Lead

    1 week ago

    Brisbane, Queensland, Australia beBeeData Full time $120,000 - $150,000

    Job DescriptionAs a Senior Data Engineer, you will be responsible for designing and implementing end-to-end data pipelines and platforms using Azure and Microsoft Fabric. You will work closely with cross-functional teams to ensure seamless project execution and successful outcomes.">Key ResponsibilitiesDesign and implement scalable data platforms using Azure...

  • Data Engineering Expertise Lead

    15 hours ago

    Brisbane, Queensland, Australia beBeeData Full time $126,444 - $179,625

    Job OverviewThe role of Senior Data Engineer involves delivering and maintaining data assets, enhancing data quality, and driving business outcomes. Key responsibilities include leading solution development, data flow design, and data profiling to deliver value. This includes providing technical leadership, influencing team performance, mentoring junior...

  • Lead Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Endava Full time US$125,000 - US$175,000 per year

    Company Description Technology is our how. And people are our why. For over two decades, we have been harnessing technology to drive meaningful change.By combining world-class engineering, industry expertise and a people-centric mindset, we consult and partner with leading brands from various industries to create dynamic platforms and intelligent digital...