Data Engineer

2 weeks ago

Sydney, New South Wales, Australia Pathway Search Full time $180,000 - $200,000 per year

Data Engineer

Location: Sydney (Hybrid)

Salary: $150-$180k + Shares

About the Role We are seeking a hands-on Data Engineer to design, build, and operate reliable data ingestion and transformation pipelines on a modern Python, Databricks, and AWS stack. This role will enable analytics and machine learning teams by delivering trustworthy, well-modeled, and observable data.

Key Responsibilities

  • Data Platform & Pipelines – Design, build, and maintain scalable batch and streaming pipelines in Databricks (Spark, Delta Lake, Workflows/Jobs) and AWS.
  • Ingestion & Integration – Develop resilient connectors from APIs, databases, SaaS tools, and event streams (Kafka/Kinesis). Manage schema drift, backfills, and reprocessing.
  • Modeling & Storage – Implement Lakehouse/Medallion (bronze/silver/gold) layers; design efficient Delta/Parquet tables with partitioning, Z-ORDERing, and ACID guarantees.
  • Orchestration & Automation – Manage scheduling/orchestration (Databricks Workflows, Airflow), deployment automation, and parameterization.
  • Data Quality & Testing – Enforce contracts/validations, define SLAs for freshness and completeness, manage lineage/metadata (Unity Catalog).
  • Performance & Cost Optimisation – Tune Spark jobs, optimize IO and shuffles, and implement monitoring/cost controls.
  • Security & Governance – Apply IAM, encryption, secrets management, and ensure compliant handling of sensitive data.
  • Observability & Reliability – Implement monitoring, alerts, runbooks, and contribute to on-call support for data services.
  • Data Services – Deliver curated datasets via APIs or query layers, with clear documentation.
  • Collaboration & ML Enablement – Partner with analysts, ML engineers, and software teams to support feature pipelines and model deployment.

About You Required Skills & Experience

  • Bachelor's/Master's in Computer Science, Data Science, Engineering, or equivalent experience.
  • Advanced knowledge of Python, PySpark, and SQL.
  • Deep experience with Databricks (Delta Lake, Spark, Jobs/Workflows, Unity Catalog) and data lake patterns.
  • Strong skills with AWS (S3, IAM, Lambda, Glue, Step Functions/Batch, CloudWatch).
  • Solid software engineering practices: version control (git), testing (pytest), modular design.
  • Orchestration experience (Airflow and/or Databricks Workflows).
  • Data modeling expertise (dimensional/star, Data Vault, Lakehouse).
  • Proven experience with data quality/validation and production diagnostics.
  • Ability to design/document RESTful integrations (OpenAPI familiarity).

Preferred Skills

  • Streaming & CDC (Kafka/Kinesis, Debezium/DMS).
  • Containers & runtime: Docker, Kubernetes/EKS.
  • CI/CD & IaC (GitHub Actions, Terraform/CloudFormation, dbx/Databricks Repos).
  • Experience with governance/privacy and handling PII.
  • Familiarity with dbt and BI semantic models.
  • MLOps exposure (feature stores, MLflow).

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Data#3 Full time $120,000 - $150,000 per year

    3 month initial contract + potential extensionMix of WFH and Central Sydney officeOverview:We are seeking an experienced Data Engineer to join a large-scale program for an initial 3-month contract. The role will focus on building and optimising data pipelines in an Azure environment, working closely with technical teams to deliver efficient, high-quality...

  • Data Engineer

    2 days ago

    Sydney, New South Wales, Australia Talent Insights Group Full time $100,000 - $140,000 per year

    Data Engineer ( SQL / GCP / BigQuery )6-Month Contract | Possibility of ExtensionSydney / Hybrid$ a day inc super DOEWe're partnering with a leading global enterprise client to hire a Mid-Level GCP Data Engineer on an initial 6-month contract with the possibility of extension. This is an exciting opportunity to work on cloud-first, enterprise-scale data...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    OverviewOur NSW Government client are hiring for a highly capable Data Engineer to join the organisation. The Data Engineer is responsible for designing and building scalable, reusable data pipelines and transformations that power the organisation's cloud-based data platform. This role focuses on delivering high-quality, production-ready solutions that...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    OverviewOur NSW Government client are hiring for a highly capable Data Engineer to join the organisation. The Data Engineer is responsible for designing and building scalable, reusable data pipelines and transformations that power the organisation's cloud-based data platform. This role focuses on delivering high-quality, production-ready solutions that...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Talenza Full time $80,000 - $120,000 per year

    Our NSW Government client are hiring for a highly capable Data Engineer to join the organisation. The Data Engineer is responsible for designing and building scalable, reusable data pipelines and transformations that power the organisation's cloud-based data platform. This role focuses on delivering high-quality, production-ready solutions that support...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia UCentric Full time $120,000 - $180,000 per year

    Data Engineer | Hybrid (Sydney)We are partnering with a fast-growing technology organisation that is building out its data capability and looking for talented Data Engineers to join their team. This is a great environment if you're motivated by solving complex problems, working with cutting-edge tools, and growing your career with support and structure...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia UCentric Full time $100,000 - $140,000 per year

    Our ClientWe are working with a rapidly growing firm that has already grown to over 100 team members in just over a year and continues to scale. The company partners with a wide range of clients to deliver modern, cloud-based data solutions. What sets this business apart is its thoughtful team structure. Engineers are supported by Engagement Leads who take...

  • Data Engineers

    4 weeks ago

    Sydney, New South Wales, Australia NOVON Full time

    OverviewData Engineers - Azure at NOVONNovon is a boutique Tech Consulting Company with Blue-chip (ASX50) clients. Join a connected Data and Engineering Community. 10 years delivering successful data outcomes.Are you a Data Engineering junky, immersed in data-driven projects that help organisations modernise their platforms and build scalable analytics...

  • Data Engineers

    4 weeks ago

    Sydney, New South Wales, Australia NOVON Full time

    OverviewData Engineers - Azure at NOVONNovon is a boutique Tech Consulting Company with Blue-chip (ASX50) clients. Join a connected Data and Engineering Community. 10 years delivering successful data outcomes.Are you a Data Engineering junky, immersed in data-driven projects that help organisations modernise their platforms and build scalable analytics...

  • Data Engineers

    2 weeks ago

    Sydney, New South Wales, Australia HiTech Personnel Full time $120,000 - $150,000 per year

    Fantastic opportunity within a Federal Government DepartmentInitial 12 month contract + 12 month extensionHybrid role based in NSW & VICOne of Australia's principal Federal Government departments is looking to engage skilled Data Engineersto support the ongoing development and maintenance of its national cloud-based data infrastructure. The successful Data...