▷ Urgent Principal Data Engineer

4 weeks ago

Perth, Australia Kirra Services Full time

Overview

The Principal Data Engineer is responsible for building, testing, and operating an enterprise data platform.

The engineer shall ensure that the platform is developed and deployed with a secure-by-design approach, delivering robust, thoroughly tested, and maintainable solutions. Moreover, the Platform Data Engineer shall provide support, security reviews, and technical reviews for delivery data engineers working on the platform.

Mandatory Requirements

- The engineer shall have experience integrating with timeseries source systems: Honeywell Plant Historian Database, OSI Pi
- The engineer shall have extensive experience in Authentication mechanisms including but not limited to (OAuth 2.0, OIDC, Microsoft Entra, Key Pair Authentication, Certificate based authentication, SAML based SSO).
- Use of AWS services for cloud-based platform tooling infrastructure including but not limited to: Lambda,ECS,MSK,RDS,EC2, Secrets Manager, ALB, Cloud Watch, Event Bridge

Required Professional and Technical Expertise

- Development of standard patterns for data pipelines and workflows utilising Streamsets, Kestra, dbt, Git.
- APIs / Integration for data access and manipulation.
- Development of API’s and Web Applications to support the management of the Platform (Security, Sandbox creation etc.)
- Experience in Terraform for AWS and Azure deployments.
- Python scripting for data common processing and automation tasks.
- CLI development to automate common platform tasks.
- Data storage design and processing solutions employing Snowflake.
- Quality assurance and testing

Desired Attributes and Skills

- Provide technical support to delivery data engineers working on the platform
- Conduct security reviews for data engineering projects to ensure compliance with standards.
- Offer technical reviews and feedback to enhance the quality of deliverables.
- Testing and Quality Assurance experience

Platform Data Engineer responsibilities and AWS/Data Platform capabilities

A data engineer with expertise in AWS toolset advises on, develops, and maintains data engineering solutions on the AWS Cloud ecosystem. They design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis. Additionally, they create data layers on AWS RedShift, Aurora, and DynamoDB. The data engineer also migrates data using AWS DMS and is proficient with various AWS Data Platform components, including S3, RedShift, RedShift Spectrum, AWS Glue with Spark, AWS Glue with Python, Lambda functions with Python, AWS Glue Catalog, and AWS Glue Databrew. They are experienced in developing batch and real-time data pipelines for Data Warehouse and Datalake, utilising AWS Kinesis and Managed Streaming for Apache Kafka. They are also proficient in using open-source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on the AWS Platform. The data engineer schedules and manages data services on the AWS Platform, ensuring seamless integration and operation of data engineering solutions.

#J-18808-Ljbffr

  • Principal Data

    2 weeks ago

    Perth, Australia BHP Group Full time

    Press Tab to Move to Skip to Content Link Select how often (in days) to receive an alert: All BHP employees and contractors please apply using your BHP credentials / email address. Please click on 'BHP Employee / Contractor Login' button on thetop right of this page to apply. Principal Data & AI Platform Engineer Date: 30 Sept 2025 Job Posting End Date:...

  • ▷ (Urgent Search) Principal Tailings Engineer

    4 weeks ago

    Perth, Australia Geosyntec Consultants, Inc. Full time

    Overview Geosyntec is growing in Australia. We now have over 110 practitioners across the country, with offices in Perth, Brisbane, Sydney, Melbourne and Gold Coast. In 2022, Red Earth Engineering joined the Geosyntec family of companies, bringing with them some of the very best expertise in the mine tailings and water dam sectors. We are now the tailings...

  • Principal Data Scientist

    9 hours ago

    Perth, Western Australia fmgl Full time $120,000 - $180,000 per year

    About Us Fortescue is both a proud West Australian company and a global green solutions business. We are recognised for our culture, innovation and industry-leading development of infrastructure, mining assets and green energy initiatives. Our Opportunity Work Location: Perth – Fortescue's Perth office is located on the traditional lands of the Whadjuk...

  • Backup Engineer

    2 days ago

    Perth, Western Australia Data#3 Full time $90,000 - $120,000 per year

    Data#3 is seeking an experiencedBackup Engineer / Specialistfor a6-month contractwith one of our key clients, basedon-site in Perth.In this role, you'll be responsible for managing and optimising anenterprise backup environmentusingDELL PowerProtect Data Manager. You'll play a critical part in ensuring data protection, reliability, and compliance across the...

  • Senior Principal Structural Engineer

    2 weeks ago

    Perth, Western Australia McDermott Full time $120,000 - $250,000 per year

    Job OverviewThe Senior Principal Structural Engineer is tasked with solving unique and complex problems that have a broad impact on the business. They must proactively identify and solve the most complex problems that impact the management and direction of the business. The Senior Principal Structural Engineer is considered a specialist in the field of...

  • Senior Principal Structural Engineer

    2 weeks ago

    Perth, Western Australia McDermott International, Ltd Full time $200,000 - $250,000 per year

    Job OverviewJOB DESCRIPTIONThe Senior Principal Structural Engineer is tasked with solving unique and complex problems that have a broad impact on the business. They must proactively identify and solve the most complex problems that impact the management and direction of the business. The Senior Principal Structural Engineer is considered a specialist in the...

  • Principal Engineering Technician

    3 days ago

    Perth, Australia Western Power Full time

    **Principal Engineering Technician**: - Location: Perth Office - CBD, WA, AU- 13 Nov 2024**Principal Engineering Technician: Permanent - Full-Time / Perth CBD** We currently have an exciting opportunity for a Principal Engineering Technician at our Perth CBD Office on a Permanent Full-Time basis. You will be joining Western Power’s Asset Management...

  • Principal Process Engineer

    1 week ago

    Perth, Western Australia MINCORE PTY LTD Full time $120,000 - $180,000 per year

    Principal Process EngineerType: ContractorLocation: preferably Perth (WA)Reporting to: Study ManagerRoleThis is a technical leadership role responsible for the development of an optimised flowsheet to PFS level requirements for a WA based ionic clay hosted rare earths project.It includes test work review and interface and coordination, process definition,...

  • Principal Software Engineer

    8 hours ago

    Greater Perth Area, Australia Verse Full time $120,000 - $170,000 per year

    About the RoleA Principal Software Engineer position is available with a product-based business in the asset management sector. This is a newly created role designed to add technical seniority and coverage across a busy, growing engineering team.You will work closely with the Head of Software to ensure projects are well architected, maintain technical...

  • Principal - Dataops

    2 weeks ago

    Perth, Australia Water Corporation Full time

    **Principal - DataOps** **-** **0Z8G** **Primary Location** WA-Perth - Inner-John Tonkin Water Centre - Leederville **Organization** Data & Analytics - **Join the newly restructured and fast growing Data & Analytics business unit**: - **New role offers you the opportunity to make your mark**: - **Permanent role with flexible working arrangements...