AWS Data Engineer

1 day ago

Sydney, New South Wales, Australia Recmatrix Consulting Full time $80,000 - $120,000 per year

Job Title: AWS Data Engineer

Role: Contract

Location: Sydney, Australia

Experience: 6-10 years

About the Position:

The
Data Engineer
will play a crucial role in designing, developing, and executing data processing and migration strategies. This role involves utilizing various big data technologies and AWS services to support a smooth transition from on-premise Hadoop to a cloud-based environment. Responsibilities include managing data migration, optimization, and integration tasks to deliver robust and scalable data solutions.

Key Tasks:

  • Create and Execute Data Migration Strategies:
    Transition data from on-premise Hadoop to AWS.
  • Develop and Optimize Spark Jobs:
    For data processing on AWS EMR.
  • Manage S3 Storage Solutions:
    Oversee data lake architecture and maintenance.
  • Configure and Oversee AWS Glue:
    For ETL processes and data cataloging.
  • Ensure Data Quality and Integrity:
    Maintain high standards throughout the migration.
  • Work with Stakeholders:
    Gather and document data requirements.
  • Handle Data Mapping and Transformation:
    As part of the migration strategy.
  • Implement AWS Lake Formation:
    Manage data governance and security.
  • Resolve Data Migration Issues:
    Troubleshoot and address performance bottlenecks.
  • Monitor and Enhance Data Workflows:
    Optimize for performance and cost efficiency.
  • Document Data Processes and Architecture:
    Develop and maintain detailed documentation.

Essential Skills and Experience:

  • Bachelor's or Master's Degree:
    In
    Engineering, Computer Science, or a related
    discipline.
  • Experience:
    6-10 years working with technologies such as
    Spark, Oracle 11g, Hive, HDFS, Hadoop, and AWS services.
  • 4-6 years
    of experience specifically with
    data migration and big data technologies.
  • Proficient in Apache Spark and AWS EMR.
  • Extensive experience with AWS services:
    Including S3, Glue, and Lake Formation.
  • Skilled in HDFS, Hive, and data migration strategies.
  • Strong grasp of data processing and optimization techniques.
  • Familiarity with building reusable ETL frameworks/ETL development experience using Spark and Scala.
  • Prior experience in migrating ETL jobs to AWS environments.
  • Comprehensive understanding of the ETL lifecycle, including running, debugging, testing, reconciling, and deploying jobs in an AWS environment.

  • AWS Data Engineer

    3 days ago

    Sydney, New South Wales, Australia FR Consultancy Full time $120,000 - $180,000 per year

    Role:AWS Data EngineerType:ContractLocation:SydneyStatus:Valid AUS visa holders.JD 1: Senior Data Engineer position here is the skills summary (the AWS data stack is must have)Sound understanding of Data modelling conceptsStrong knowledge of SQLAWS Data Stack: S3,RDS, Redshift, Iceberg and Glue (proficient)Programming languages: Python/Spark (working...

  • Data Engineer AWS

    1 day ago

    Sydney, New South Wales, Australia The Nudge Group Full time $120,000 - $180,000 per year

    We are after a Senior AWS Data Engineer to work with a consulting firm on a top client.Experience required:A hands-on engineer with deep experience working with a variety of AWS services for ingestion, curation, storage, orchestration, data management, analytics and distribution.Good knowledge of data modelling and how to structure data within a data...

  • AWS Data Engineer

    3 days ago

    Sydney, New South Wales, Australia CareCone Group Full time $120,000 - $180,000 per year

    Title: AWS Data EngineerLocation: SydneyPermanent RoleKey Responsibilities:Design and implement data pipelines for seamless data ingestion.Utilize AWS technologies, including S3, Glue, and Lambda, for effective data management.Collaborate with cross-functional teams to enhance data processes using Python, PySpark, and SQL.Implement CI/CD deployments for...

  • AWS Data Engineer

    3 days ago

    Sydney, New South Wales, Australia CareCone Group Full time $150,000 - $200,000 per year

    Role: AWS Data Engineer (Snowflake)Location: SydneyJob Description:Analyse and organise raw data.Must have 10+ years of experience.Design develop and maintain data pipelines using AWS technologies such as AWS Glue, Airflow, Redshift and databricks.Implement data modelling techniques to optimize data and storage and retrieval process.Must have knowledge of...

  • AWS Data Platform Engineer

    7 days ago

    Sydney, New South Wales, Australia Maltem Australia Full time $120,000 - $180,000 per year

    About Maltem AustraliaMaltem Australia is a boutique consultancy specialising in data and analytics, part of the Maltem Group a global consulting firm with over 25 years of experience and operations in 12+ countries.In Australia, we partner with leading organisations across financial services, technology, and government to design and implement modern data...

  • AWS Data Engineer

    7 days ago

    Sydney, New South Wales, Australia Synechron Full time $100,000 - $120,000 per year

    $950 plus GST ( must have ABN)Hybrid workingInitial 6 months contractWe AreWe are At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron's progressive technologies and optimization strategies span...

  • Data Center Technical Operations Engineer

    1 week ago

    Sydney, New South Wales, Australia Amazon Web Services (AWS) Full time $80,000 - $120,000 per year

    DescriptionAWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we're the people who keep the cloud running. We support all AWS data centers and all of the servers, storage, networking, power, and cooling equipment that ensure our customers have continual access to the innovation...

  • AWS Data Platform

    3 days ago

    Sydney, New South Wales, Australia NOVON Full time $120,000 - $180,000 per year

    Are you passionate about buildingresilient, scalable data platforms? Do you thrive in high-impact environments whereautomation, reliability, and uptimeare critical? AtNOVON, we are looking for an AWS Data Platform Engineer to join our team. This is a pivotal role where you'll combineDevOps engineering, data platform expertise, and AWS cloud servicesto build...

  • AWS Data Engineer

    3 days ago

    Sydney, New South Wales, Australia Avance Consulting Full time $120,000 - $180,000 per year

    Job DescriptionJD 1: Senior Data Eng position here is the skills summary (the AWS data stack is must have)Sound understanding of Data modelling conceptsStrong knowledge of SQLAWS Data Stack: S3,RDS, Redshift,Iceberg and Glue (proficient)Programming languages: Python/Spark (working knowledge)Data warehousing / ETL : Glue,EMR or Abinitio (working...

  • Lead Data Engineer – AWS

    2 weeks ago

    Sydney, New South Wales, Australia Bluefin Resources Pty Limited Full time $80,000 - $140,000 per year

    Join a high-performing Data team that's driving a large-scale transformation program within the banking and financial services industry. This is an excellent opportunity for an ambitious and technically strong Data Engineer to play a lead role in shaping a modern, cloud-based data platform. You'll be joining a collaborative, forward-thinking environment...