Principle Data Engineer

2 days ago


Perth, Western Australia Paxus - Technology + Digital Talent Full time
  • 6 month initial contract
  • Perth based; hybrid - 3 days onsite
  • Multiple roles *3
  • AWS Toolset

Tasks to be performed on a daily basis:
The Principal Data Engineer shall be responsible for the building, testing, and operation of an enterprise data platform.

The engineer shall ensure that the platform is developed and deployed with a secure-by-design approach, delivering robust, thoroughly tested, and maintainable solutions. Moreover, the Platform Data Engineer shall provide support, security reviews, and technical reviews for delivery data engineers working on the platform.

A data engineer with expertise in AWS toolset advises on, develops, and maintains data engineering solutions on the AWS Cloud ecosystem. They design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis. Additionally, they create data layers on AWS RedShift, Aurora, and DynamoDB.

The data engineer also migrates data using AWS DMS and is proficient with various AWS Data Platform components, including S3, RedShift, RedShift Spectrum, AWS Glue with Spark, AWS Glue with Python, Lambda functions with Python, AWS Glue Catalog, and AWS Glue Databrew. They are experienced in developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. They are also proficient in using open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform.

The data engineer schedules and manages data services on the AWS Platform, ensuring seamless integration and operation of data engineering solutions.

Mandatory Requirements:

  • Development experience with Go, SQL, C#, .net, JavaScript, shell scripts & container platforms like Docker.
  • The engineer shall have experience integrating with timeseries source systems: Honeywell Plant Historian Database, OSI Pi.
  • The engineer shall have extensive experience in Authentication mechanisms including but not limited to (OAuth 2.0, OIDC, Microsoft Entra, Key Pair Authentication, Certificate based authentication, SAML based SSO).
  • Use of AWS services for cloud-based platform tooling infrastructure including but not limited to: Lambda, ECS, MSK, RDS, EC2, Secrets Manager, ALB, Cloud Watch, Event Bridge.

Required Professional and Technical Expertise:

  • Development of standard patterns for data pipelines and workflows utilizing Streamsets, Kestra, dbt, Git.
  • APIs / Integration for data access and manipulation.
  • Development of APIs and Web Applications to support the management of the Platform (Security, Sandbox creation etc.).
  • Experience in Terraform for AWS and Azure deployments.
  • Python scripting for data common processing and automation tasks.
  • CLI development to automate common platform tasks.
  • Data storage design and processing solutions employing Snowflake.
  • Quality assurance and testing.

Desired Attributes and Skills:

  • Provide technical support to delivery data engineers working on the platform.
  • Conduct security reviews for data engineering projects to ensure compliance with standards.
  • Offer technical reviews and feedback to enhance the quality of deliverables.
  • Testing and Quality Assurance experience.

To be considered for the role, click the 'apply' button or for more information about this and other opportunities please contact Gizaille Shaheen on 03 86804305 or email: Gshaheen@paxus.com.au and quote the above job reference number.

#J-18808-Ljbffr

  • Perth, Western Australia VenuesWest Full time

    About the RoleVenuesWest is seeking a Performance Data Engineer to join our team. In this role, you'll be responsible for designing and implementing performance data solutions. This includes developing and maintaining data pipelines, creating reports and dashboards, and ensuring data quality and security.Key Responsibilities:Design and implement performance...


  • Perth, Western Australia Tbwa ChiatDay Inc Full time

    About VGWVGW is a leading interactive entertainment company that harnesses technology and creativity to deliver world-class, free-to-play games.We have an exciting opportunity to join our Technology team in Australia and are currently seeking a highly skilled Director of Data Architecture and Engineering to lead our data engineering efforts.As the Director...

  • Data Engineer

    2 weeks ago


    Perth, Western Australia Tideri Jobbörse Full time

    Do you want a job that looks different every day? Have you got an unquenchable passion for data and problem solving? Are you aching to jump into a work environment where you will work with some of the keenest minds in the industry? We've got a career you will enjoy At InterWorks, we like our work with a hefty side of play. Our team of high achievers is...


  • Perth, Western Australia Tbwa ChiatDay Inc Full time

    VGW is an interactive entertainment company, harnessing technology and creativity to deliver world-class, free-to-play games. We have an exciting opportunity to join our Technology team in Australia and are currently looking for a Head of Data Engineering to join the team. As the Head of Data Engineering at VGW, you will lead the strategy to deliver...

  • Data Engineer

    15 hours ago


    Perth, Western Australia InterWorks Full time

    Do you want a job that looks different every day? Have you got an unquenchable passion for data and problem solving? Are you aching to jump into a work environment where you will work with some of the keenest minds in the industry? We've got a career you will enjoyAt InterWorks, we like our work with a hefty side of play. Our team of high achievers is...

  • Data Engineer

    2 days ago


    Perth, Western Australia Robert Walters Full time

    Our client, a leading player in the financial services sector, is seeking a Data Engineer to play a pivotal role to leverage technology.This is an opportunity to revolutionise financial wellness by merging innovative technology with human expertise. As a key contributor, you will build and maintain their data infrastructure, ensuring data is reliable,...

  • Data Engineer

    2 days ago


    Perth, Western Australia Whizdom Full time

    Whizdom Perth, Western Australia, AustraliaState Manager - taking the Whizdom success story further into NSW, QLD and VICAbout Our ClientOur client is a globally recognized consulting house with a strong reputation for delivering innovative solutions across various sectors, including technology, energy, and finance. With a presence in multiple countries,...


  • Perth, Western Australia Fortescue Full time

    At Fortescue, we are committed to leading the world in decarbonisation and sustainability. As a key player in our Heavy Industry projects, you will play a crucial role in developing and testing technologies to power our mining fleet and rail locomotives.We are seeking an experienced Data Engineer to join our team at our new Redcliffe and Hazelmere R&D...

  • Cloud Data Engineer

    3 days ago


    Perth, Western Australia Fortescue Full time

    Fortescue is driving innovation in heavy industry through our Heavy Industry projects, focusing on green energy and carbon neutrality. We are seeking a talented Data Engineer to join our team at our Redcliffe and Hazelmere R&D facilities.In this role, you will work collaboratively with a multidisciplinary engineering team to develop and test technologies...


  • Perth, Western Australia VenuesWest Full time

    Position number: 019604Work type: Permanent - Full TimeLocation: Perth CBDThis is a permanent, full-time vacancy.About JusticeWhen you join the Department of Justice, you become part of a people-focused organisation, working hard to make our community a fairer and safer place to live. We offer many different employment opportunities in locations around...


  • Perth, Western Australia VitalTrace Full time

    About the Role:VitalTrace is seeking an experienced Data Engineer to lead our data engineering initiatives. As a key member of our team, you will be responsible for designing, building, and maintaining data pipelines, developing applications, and implementing data management solutions.Main Responsibilities:Develop and maintain efficient data pipelines to...


  • Perth, Western Australia Paxus - Technology + Digital Talent Full time

    About the PositionThe AWS Data Platform Engineer will be responsible for designing, building, and operating data engineering solutions on the AWS Cloud ecosystem. The successful candidate will have expertise in AWS toolset and advise on, develops, and maintains data engineering solutions.Key ResponsibilitiesDesign, build, and operate batch and real-time data...


  • Perth, Western Australia NXTGIG Full time

    NXTGIG is a leading technology company based in Perth, Australia, that delivers innovative AI solutions.AI Engineer PositionWe are seeking a highly skilled AI Engineer with expertise in Python to join our team. As an AI Engineer, you will design, develop, and deploy AI-driven solutions that leverage advanced machine learning techniques and algorithms. You...


  • Perth, Western Australia Robert Walters Full time

    This is an exciting opportunity for a Cloud Data Engineer Leader to join Robert Walters and play a pivotal role in revolutionising financial wellness. As a key contributor, you will design, develop, and maintain our data infrastructure, ensuring data is reliable, secure, and accessible for driving insights and delivering better wealth experiences.Key...


  • Perth, Western Australia Zenith Search Full time

    Engineering Cost Controller WantedSeeking a skilled Engineering Cost Controller to join our team. Salary: AUD130000 - AUD160000 per annum + SuperAbout the OpportunityThis is an excellent opportunity for a seasoned professional to take on a challenging role in cost management. As an Engineering Cost Controller, you will be responsible for overseeing the...


  • Perth, Western Australia Paxus - Technology + Digital Talent Full time

    About the RoleThe Senior Data Engineering Specialist will be responsible for building, testing, and operating an enterprise data platform on AWS. This role requires a secure-by-design approach to develop robust, thoroughly tested, and maintainable solutions.Key ResponsibilitiesDesign, build, and operate batch and real-time data pipelines using AWS services...


  • Perth, Western Australia Business in Heels International Pty Full time

    Becoming part of Business in Heels International Pty means being part of a pioneering organization that pushes boundaries through innovative science and technology. As a leading research organization, we deliver solutions with real impact, transforming lives and shaping the future.We are seeking a highly skilled software engineer to join our team in Perth,...


  • Perth, Western Australia NXTGIG Full time

    NXTGIG Perth, Western Australia, AustraliaAI Engineer with PythonNXT GIG is looking for an AI Engineer with expertise in Python to join our dynamic tech team. In this role, you will be tasked with designing, developing, and deploying AI-driven solutions that leverage advanced machine learning techniques and algorithms. Collaborating closely with data...


  • Perth, Western Australia Airswift Full time

    1x Senior Project Telecommunications Engineer Location: Perth The company: Major EPC Player Benefits: Competitive rate with long-term prospect Indicative assignment length: Initial 12-month contract (subject to agreement) The Company Our client is a well-known consultancy with multiple offices across the globe.As a result of ongoing business growth, they...


  • Perth, Western Australia Jacobs Full time

    Jacobs is a global company dedicated to delivering technical solutions across diverse clients. We're seeking a seasoned Principal Systems Engineer to join our Perth office and support the ANZ Rail & Transit team in delivering technical solutions.The role requires strong client relationships, clear technical direction, and exceptional project delivery...