Lead Data Engineer

1 week ago

Sydney, New South Wales, Australia beBeeDataEngineer Full time $180,000 - $230,000

As a data engineer in our data platform team, you will have the opportunity to build, optimize and grow one of the world's largest data platforms that directly supports millions of users.
You'll gain hands-on experience on all kinds of systems in the data platform ecosystem.

Job Description:

We are seeking an experienced data engineer who can design, implement and maintain reliable, scalable, robust and extensible big data systems. You will work closely with engineers, product managers and data scientists to understand data needs, represent key data insights in a meaningful way and establish solid design and best engineering practices for engineers and non-technical people.

Responsibilities:
  • Design, implement and maintain big data systems that support core products and business
  • Design, build and maintain data pipelines utilizing optimal ETL patterns, frameworks and query techniques
  • Extract data from various sources such as APIs, HIVE tables and other structured and unstructured data sources to process and store large volumes of data ensuring data accuracy, consistency and security
  • Implement and monitor quality control measures to ensure data accuracy, completeness and consistency
  • Create and maintain technical documentation to ensure efficient and effective data management and analysis
  • Optimize pipelines, dashboards, frameworks and systems to facilitate easier development of data artifacts
Required Skills and Qualifications:

Minimum qualifications include a Bachelors degree in Statistics, Computer Science or a related technical field involving software or systems engineering, or equivalent practical experience. You must have 1+ years of experience working with data analytics and data engineering, including experience with data cleaning and preprocessing, data ingestion and ETL, data analysis and dashboard development.

Preferred Qualifications:
  • Proficiency in distributed data processing using Big Data technologies like Spark/Scala, Python, Hadoop/HDFS/AWS/S3 and Kafka
  • Proficiency in data modeling, data design, SQL and databases
  • 3+ years of Experience with Big Data technologies (Hadoop, M/R, Hive, Spark, Metastore, Presto, Flume, Kafka, ClickHouse, Flink etc)
  • Strong background in algorithms and data structures
  • Experience working with PII and GDPR data
  • Ability to communicate effectively, both written and verbal, with technical and non-technical partners
  • Ability to deliver consistent high quality results while working in a fast paced environment
Others:

Our organization follows a hybrid work schedule that requires employees to work in the office 3 days a week. We regularly review our hybrid work model and the specific requirements may change at any time.

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Data#3 Full time $90,000 - $120,000 per year

    3 month initial contract + potential extensionMix of WFH and Central Sydney officeOverview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Data#3 Full time

    Overview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality data solutions.Responsibilities:Design, build, and maintain ETL pipelines using...

  • Data Engineer Consultant

    3 weeks ago

    Sydney, New South Wales, Australia NTT DATA, Inc. Full time

    Join to apply for the Data Engineer Consultant role at NTT DATA, Inc.3 days ago Be among the first 25 applicantsJoin to apply for the Data Engineer Consultant role at NTT DATA, Inc.Get AI-powered advice on this job and more exclusive features.Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for...

  • Data Engineering Lead

    1 week ago

    Sydney, New South Wales, Australia Tribus Full time $125,000 - $175,000 per year

    Data Engineering Lead We're looking for a Data Engineering Lead to lead the build-out of scalable data infrastructure for real-time analytics and reporting. This is a leadership-first role (60%) with continued technical delivery (40%). In This Role, You Will:Lead a growing team of data engineers growing to 6. Build and manage streaming data pipelines...

  • Solar Design Engineer

    1 week ago

    Sydney, New South Wales, Australia Electrical Engineer Full time

    OverviewHarvey Norman Commercial Division specialises in the supply of goods & services to project builders, developers, architects and the mining industry. Our clients depend upon Harvey Norman Commercial to supply the best range, the best service and the best prices.We have an exciting opportunity at our Taren Point office for a motivated and driven Full...

  • Solar Design Engineer

    1 week ago

    Sydney, New South Wales, Australia Electrical Engineer Full time

    OverviewHarvey Norman Commercial Division specialises in the supply of goods & services to project builders, developers, architects and the mining industry. Our clients depend upon Harvey Norman Commercial to supply the best range, the best service and the best prices.We have an exciting opportunity at our Taren Point office for a motivated and driven Full...

  • Lead Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Talenza Full time $230,000 - $240,000 per year

    Lead Data Engineer Sydney, Australia $230,000 (including Super) + Bonus About the Role: Join a high-performing Data Team at a leading financial services institution to lead regulatory initiatives and drive innovation in data engineering, utilizing AI and automation to enhance platform efficiency. Key Responsibilities:Lead the development and...

  • Data Engineering Lead

    5 days ago

    Sydney, New South Wales, Australia beBeeEngineering Full time $123,456 - $234,567

    Lead Data Engineering Position This role is for a senior data engineer who can lead the design and delivery of robust data pipelines. The ideal candidate will have experience in leading cross-functional teams and guiding them through the full lifecycle of a project from discovery to production. They must also be able to translate technical complexity into...