Data Engineer

1 day ago


Perth, Western Australia Whizdom Full time

Whizdom Perth, Western Australia, Australia

State Manager - taking the Whizdom success story further into NSW, QLD and VIC

About Our Client

Our client is a globally recognized consulting house with a strong reputation for delivering innovative solutions across various sectors, including technology, energy, and finance. With a presence in multiple countries, they offer a diverse range of services and have a proven track record of success. Their unique selling points include a commitment to excellence, a collaborative work environment, and access to cutting-edge technology.

About the Role

We are seeking a Principal Data Engineer to join our client's dynamic team. In this role, you will be responsible for building, testing, and operating an enterprise data platform.

Key criteria for this position include:

  • Experience integrating with timeseries source systems such as Honeywell Plant Historian Database and OSI Pi.
  • Extensive experience in authentication mechanisms including OAuth 2.0, OIDC, Microsoft Entra, Key Pair Authentication, Certificate-based authentication, and SAML-based SSO.
  • Proficiency in using AWS services for cloud-based platform tooling infrastructure, including Lambda, ECS, MSK, RDS, EC2, Secrets Manager, ALB, CloudWatch, and EventBridge.
  • Development of standard patterns for data pipelines and workflows utilizing Streamsets, Kestra, dbt, and Git.
  • APIs/Integration for data access and manipulation.
  • Development of APIs and Web Applications to support the management of the platform (Security, Sandbox creation, etc.).
  • Experience in Terraform for AWS and Azure deployments.
  • Python scripting for data processing and automation tasks.
  • CLI development to automate common platform tasks.
  • Data storage design and processing solutions employing Snowflake.
  • Quality assurance and testing.

The Successful Candidate

The ideal candidate will have the following credentials, qualifications, and experience:

  • Proven experience as a Data Engineer with expertise in AWS toolset.
  • Ability to design, build, and operate batch and real-time data pipelines using AWS services.
  • Proficiency in developing data layers on AWS RedShift, Aurora, and DynamoDB.
  • Experience in migrating data using AWS DMS.
  • Familiarity with open-source technologies like Apache Airflow and dbt, Spark/Python or Spark/Scala on AWS Platform.
  • Strong technical support skills for delivery data engineers working on the platform.
  • Conducting security reviews for data engineering projects to ensure compliance with standards.
  • Offering technical reviews and feedback to enhance the quality of deliverables.
  • Passionate, diligent, hardworking, with exceptional communication skills.

What is on Offer

This opportunity offers the chance to work in a prestigious consulting house with amazing end clients. You will benefit from competitive contracting rates, access to world-class technology, and learning and development opportunities. You will also have the chance to work with talented managers and colleagues in a collaborative and innovative environment.

Seniority level

Mid-Senior level

Employment type

Contract

Job function

Information Technology

Industries

IT Services and IT Consulting, Wholesale Petroleum and Petroleum Products, and Mining

#J-18808-Ljbffr
  • Data Engineer

    1 day ago


    Perth, Western Australia Robert Walters Full time

    Our client, a leading player in the financial services sector, is seeking a Data Engineer to play a pivotal role to leverage technology.This is an opportunity to revolutionise financial wellness by merging innovative technology with human expertise. As a key contributor, you will build and maintain their data infrastructure, ensuring data is reliable,...


  • Perth, Western Australia Paxus - Technology + Digital Talent Full time

    6 month initial contractPerth based; hybrid - 3 days onsiteMultiple roles *3AWS ToolsetTasks to be performed on a daily basis:The Principal Data Engineer shall be responsible for the building, testing, and operation of an enterprise data platform.The engineer shall ensure that the platform is developed and deployed with a secure-by-design approach,...


  • Perth, Western Australia Fortescue Full time

    At Fortescue, we are committed to leading the world in decarbonisation and sustainability. As a key player in our Heavy Industry projects, you will play a crucial role in developing and testing technologies to power our mining fleet and rail locomotives.We are seeking an experienced Data Engineer to join our team at our new Redcliffe and Hazelmere R&D...


  • Perth, Western Australia workable - ATS Full time

    At DataDivers, we empower organizations to unlock the full potential of their data through expert guidance and innovative solutions. As a Cloud Data Engineering Manager, you will lead our team in designing and delivering world-class cloud data engineering capabilities.About the RoleAs a Cloud Data Engineering Manager, you will be responsible for:Leading a...

  • Data Engineer

    2 weeks ago


    Perth, Western Australia Tideri Jobbörse Full time

    Do you want a job that looks different every day? Have you got an unquenchable passion for data and problem solving? Are you aching to jump into a work environment where you will work with some of the keenest minds in the industry? We've got a career you will enjoy At InterWorks, we like our work with a hefty side of play. Our team of high achievers is...

  • Cloud Data Engineer

    3 days ago


    Perth, Western Australia Fortescue Full time

    Fortescue is driving innovation in heavy industry through our Heavy Industry projects, focusing on green energy and carbon neutrality. We are seeking a talented Data Engineer to join our team at our Redcliffe and Hazelmere R&D facilities.In this role, you will work collaboratively with a multidisciplinary engineering team to develop and test technologies...


  • Perth, Western Australia VitalTrace Full time

    About the Role:VitalTrace is seeking an experienced Data Engineer to lead our data engineering initiatives. As a key member of our team, you will be responsible for designing, building, and maintaining data pipelines, developing applications, and implementing data management solutions.Main Responsibilities:Develop and maintain efficient data pipelines to...

  • Data Engineer

    3 hours ago


    Perth, Western Australia InterWorks Full time

    Do you want a job that looks different every day? Have you got an unquenchable passion for data and problem solving? Are you aching to jump into a work environment where you will work with some of the keenest minds in the industry? We've got a career you will enjoyAt InterWorks, we like our work with a hefty side of play. Our team of high achievers is...


  • Perth, Western Australia Paxus - Technology + Digital Talent Full time

    About the PositionThe AWS Data Platform Engineer will be responsible for designing, building, and operating data engineering solutions on the AWS Cloud ecosystem. The successful candidate will have expertise in AWS toolset and advise on, develops, and maintains data engineering solutions.Key ResponsibilitiesDesign, build, and operate batch and real-time data...


  • Perth, Western Australia NXTGIG Full time

    NXTGIG is a leading technology company based in Perth, Australia, that delivers innovative AI solutions.AI Engineer PositionWe are seeking a highly skilled AI Engineer with expertise in Python to join our team. As an AI Engineer, you will design, develop, and deploy AI-driven solutions that leverage advanced machine learning techniques and algorithms. You...


  • Perth, Western Australia Robert Walters Full time

    This is an exciting opportunity for a Cloud Data Engineer Leader to join Robert Walters and play a pivotal role in revolutionising financial wellness. As a key contributor, you will design, develop, and maintain our data infrastructure, ensuring data is reliable, secure, and accessible for driving insights and delivering better wealth experiences.Key...


  • Perth, Western Australia Capital Executive Search Full time

    About the RoleThis is a mid-senior level, full-time position within the Engineering department. The ideal candidate will have a Bachelor's degree in Mechanical Engineering or Petroleum Engineering and a minimum of 5 years of experience in Data Centers and/or mission critical facilities design, construction, or operations.Solid knowledge of large-scale water...


  • Perth, Western Australia Paxus - Technology + Digital Talent Full time

    About the RoleThe Senior Data Engineering Specialist will be responsible for building, testing, and operating an enterprise data platform on AWS. This role requires a secure-by-design approach to develop robust, thoroughly tested, and maintainable solutions.Key ResponsibilitiesDesign, build, and operate batch and real-time data pipelines using AWS services...


  • Perth, Western Australia Business in Heels International Pty Full time

    Becoming part of Business in Heels International Pty means being part of a pioneering organization that pushes boundaries through innovative science and technology. As a leading research organization, we deliver solutions with real impact, transforming lives and shaping the future.We are seeking a highly skilled software engineer to join our team in Perth,...


  • Perth, Western Australia VenuesWest Full time

    Position number: 019604Work type: Permanent - Full TimeLocation: Perth CBDThis is a permanent, full-time vacancy.About JusticeWhen you join the Department of Justice, you become part of a people-focused organisation, working hard to make our community a fairer and safer place to live. We offer many different employment opportunities in locations around...


  • Perth, Western Australia C015 APT MANAGEMENT SERVICES PTY LIMITED Full time

    About the JobWe're looking for an experienced Data and Modelling Solutions Architect to join our OT Engineering Team Lead – Data and Modelling team.Key Responsibilities:Design and develop data management solutions to support operational technology and data science initiatives.Collaborate with cross-functional teams to identify and prioritize data science...

  • Senior Engineer

    5 days ago


    Perth, Western Australia C015 APT MANAGEMENT SERVICES PTY LIMITED Full time

    OT Engineering Team Manager PositionWe're seeking a skilled OT Engineering Team Manager to lead our Data and Modelling team.Key Responsibilities:Lead the development of team capabilities to manage Pi Historian and Online Sim modelling applications.Collaborate with cross-functional teams to identify and prioritize operational technology and data science...


  • Perth, Western Australia ROBERT WALTERS AUSTRALIA Full time

    The ideal candidate will possess a strong background in data engineering, with expertise in designing and implementing efficient data pipelines using Azure Databricks and Synapse Analytics.Key responsibilities include developing and maintaining SQL queries for data transformation and cleansing, as well as designing integration solutions using...

  • Lead Data Architect

    6 days ago


    Perth, Western Australia Fortescue Full time

    Data Engineer Leadership OpportunityFortescue is seeking a Lead Data Architect to join our team. As a technical leader, you will define the data architecture strategy, establish engineering standards, and design scalable ETL/ELT frameworks.Key ResponsibilitiesDefine and implement enterprise data architecture standards, including ETL/ELT processes, data...


  • Perth, Western Australia Commonwealth Scientific and Industrial Research Organisation Full time

    The Commonwealth Scientific and Industrial Research Organisation (CSIRO) is seeking a skilled software engineer to join our team in Perth, Western Australia. Our organisation is building one of the world's largest science facilities, with next-generation telescopes driving astrophysics research, innovation, and skills development.Job DescriptionWe are...