▷ Urgent Principal Data Engineer

1 week ago

Perth, Australia Kirra Services Full time

Overview

The Principal Data Engineer is responsible for building, testing, and operating an enterprise data platform.

The engineer shall ensure that the platform is developed and deployed with a secure-by-design approach, delivering robust, thoroughly tested, and maintainable solutions. Moreover, the Platform Data Engineer shall provide support, security reviews, and technical reviews for delivery data engineers working on the platform.

Mandatory Requirements

- The engineer shall have experience integrating with timeseries source systems: Honeywell Plant Historian Database, OSI Pi
- The engineer shall have extensive experience in Authentication mechanisms including but not limited to (OAuth 2.0, OIDC, Microsoft Entra, Key Pair Authentication, Certificate based authentication, SAML based SSO).
- Use of AWS services for cloud-based platform tooling infrastructure including but not limited to: Lambda,ECS,MSK,RDS,EC2, Secrets Manager, ALB, Cloud Watch, Event Bridge

Required Professional and Technical Expertise

- Development of standard patterns for data pipelines and workflows utilising Streamsets, Kestra, dbt, Git.
- APIs / Integration for data access and manipulation.
- Development of API’s and Web Applications to support the management of the Platform (Security, Sandbox creation etc.)
- Experience in Terraform for AWS and Azure deployments.
- Python scripting for data common processing and automation tasks.
- CLI development to automate common platform tasks.
- Data storage design and processing solutions employing Snowflake.
- Quality assurance and testing

Desired Attributes and Skills

- Provide technical support to delivery data engineers working on the platform
- Conduct security reviews for data engineering projects to ensure compliance with standards.
- Offer technical reviews and feedback to enhance the quality of deliverables.
- Testing and Quality Assurance experience

Platform Data Engineer responsibilities and AWS/Data Platform capabilities

A data engineer with expertise in AWS toolset advises on, develops, and maintains data engineering solutions on the AWS Cloud ecosystem. They design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis. Additionally, they create data layers on AWS RedShift, Aurora, and DynamoDB. The data engineer also migrates data using AWS DMS and is proficient with various AWS Data Platform components, including S3, RedShift, RedShift Spectrum, AWS Glue with Spark, AWS Glue with Python, Lambda functions with Python, AWS Glue Catalog, and AWS Glue Databrew. They are experienced in developing batch and real-time data pipelines for Data Warehouse and Datalake, utilising AWS Kinesis and Managed Streaming for Apache Kafka. They are also proficient in using open-source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on the AWS Platform. The data engineer schedules and manages data services on the AWS Platform, ensuring seamless integration and operation of data engineering solutions.

#J-18808-Ljbffr

  • Principal Data Engineer

    2 weeks ago

    Perth, Western Australia Docker Full time

    At Docker, we make app development easier so developers can focus on what matters. Our remote-first team spans the globe, united by a passion for innovation and great developer experiences. With over 20 million monthly users and 20 billion image pulls, Docker is the #1 tool for building, sharing, and running apps—trusted by startups and Fortune 100s alike....

  • ▷ (Urgent Search) Principal Tailings Engineer

    6 days ago

    Perth, Australia Geosyntec Consultants, Inc. Full time

    Overview Geosyntec is growing in Australia. We now have over 110 practitioners across the country, with offices in Perth, Brisbane, Sydney, Melbourne and Gold Coast. In 2022, Red Earth Engineering joined the Geosyntec family of companies, bringing with them some of the very best expertise in the mine tailings and water dam sectors. We are now the tailings...

  • Principal Engineer

    2 weeks ago

    Perth, Western Australia Australian Energy Market Operator (AEMO) Full time

    OverviewJoin to apply for the Principal Engineer - DER role at Australian Energy Market Operator (AEMO).Get AI-powered advice on this job and more exclusive features.AEMO at the Heart of Energy AEMO is Australia's independent energy system and market operator and system planner. AEMO's purpose is to ensure safe, reliable and affordable energy today, and...

  • Principal Engineer

    2 weeks ago

    Perth, Western Australia Australian Energy Market Operator (AEMO) Full time

    OverviewJoin to apply for the Principal Engineer - DER role at Australian Energy Market Operator (AEMO).Get AI-powered advice on this job and more exclusive features.AEMO at the Heart of Energy AEMO is Australia's independent energy system and market operator and system planner. AEMO's purpose is to ensure safe, reliable and affordable energy today, and...

  • Principal Data Scientist

    3 weeks ago

    Perth, Western Australia Fortescue Full time

    Select how often (in days) to receive an alert:Principal Data Scientist - Simulation & Techno-Economic OptimisationCountry/Region: AUState: WACity: PerthJob ID: 12343About UsFortescue is both a proud West Australian company and a global green solutions business.We are recognised for our culture, innovation and industry-leading development of infrastructure,...

  • Principal Data Scientist

    3 weeks ago

    Perth, Western Australia Fortescue Full time

    Select how often (in days) to receive an alert:Principal Data Scientist - Simulation & Techno-Economic OptimisationCountry/Region: AUState: WACity: PerthJob ID: 12343About UsFortescue is both a proud West Australian company and a global green solutions business.We are recognised for our culture, innovation and industry-leading development of infrastructure,...

  • Principal Engineer

    3 weeks ago

    Perth, Western Australia Versent Full time

    Principal Engineer (Cloud Adoption - Azure) page is loadedPrincipal Engineer (Cloud Adoption - Azure)Apply locations Perth, WA time type Full time posted on Posted 2 Days Ago job requisition id JR1266The Principal Engineer is a hands-on technical leader who drives the design, build, and optimisation of secure, scalable Microsoft Azure solutions. You'll act...

  • Principal Engineer

    3 weeks ago

    Perth, Western Australia VAU Versent Pty Ltd Full time

    Principal Engineer (Cloud Adoption - Azure) page is loadedPrincipal Engineer (Cloud Adoption - Azure)Apply locations Perth, WA time type Full time posted on Posted Today job requisition id JR1266The Principal Engineer is a hands-on technical leader who drives the design, build, and optimisation of secure, scalable Microsoft Azure solutions. You'll act as the...

  • Principal Engineer

    3 weeks ago

    Perth, Western Australia VAU Versent Pty Ltd Full time

    Principal Engineer (Cloud Adoption - Azure) page is loadedPrincipal Engineer (Cloud Adoption - Azure)Apply locations Perth, WA time type Full time posted on Posted Today job requisition id JR1266The Principal Engineer is a hands-on technical leader who drives the design, build, and optimisation of secure, scalable Microsoft Azure solutions. You'll act as the...

  • Principal Engineer

    3 weeks ago

    Perth, Western Australia Versent Full time

    Principal Engineer (Cloud Adoption - Azure) page is loadedPrincipal Engineer (Cloud Adoption - Azure)Apply locations Perth, WA time type Full time posted on Posted 2 Days Ago job requisition id JR1266The Principal Engineer is a hands-on technical leader who drives the design, build, and optimisation of secure, scalable Microsoft Azure solutions. You'll act...