Principal Data Engineer

1 month ago

Sydney, Australia Talent Insights Full time
  • Microsoft Azure, Databricks, Data Factory and Gen AI applications
  • Hybrid, flexible work
  • Opportunity to lead and shape a DE function, greenfield project

Principal / Lead Data Engineer ( Microsoft Azure / Databricks / Data Factory )
Location: Sydney (Hybrid Work)
Position Type: Full-Time, Permanent
Salary: Up to $220,000 base DOE + Superannuation

An exciting opportunity has arisen for a hands-on, proactive Principal / Lead Data Engineer to join a leading Insurance company based in Sydney.
In this role, you will be responsible for leading a team of 2-4 data engineers,
working as part of a global team on high-impact data engineering initiatives.

This is an excellent opportunity to own and shape the data engineering function, drive innovation, and play a key role in building real-time data solutions using Microsoft Azure and Generative AI technologies.

  
Key Responsibilities for the Principal / Lead Data Engineer include :

  • Lead a team of 2-4 data engineers, guiding and mentoring them to deliver high-quality data engineering solutions in a fast-paced environment.
  • Take ownership of building, optimizing, and scaling real-time data pipelines using Microsoft Azure, including Azure Databricks, Azure Data Factory, and Power BI.
  • Work with real-time data use cases: scoring data in real-time, ensuring that information is transferred and processed quickly, with an emphasis on low-latency data flow.
  • Play a key role in the development and maintenance of war boards in contact centres worldwide, enabling real-time dashboards and reporting with near-zero delays.
  • Focus on solving complex data challenges related to processing thousands of calls coming through call centers, including real-time transcriptions and passing the data for further processing.
  • Partner with the Data Science team and be involved in a journey of real-time training and AI model development, including working on training your own ChatGPT model for specific business use cases.
  • Explore and implement Generative AI applications within the data engineering function, particularly around real-time transcriptions and training models that will drive future capabilities.
  • Collaborate with the broader technology team to continuously improve data quality, security, and pipeline performance.
  • Be proactive in problem-solving, owning data engineering challenges and delivering impactful solutions under tight deadlines.

  
Key Skills & Experience required for the Principal / Lead Data Engineer include :

  • Hands-on experience with Microsoft Azure, ideally including Azure Databricks, Azure Data Factory, and Power BI.
  • Strong proficiency in Python and SQL for building scalable and efficient data pipelines.
  • Experience in real-time data processing, with a focus on minimizing delays (e.g., 1-second latency in scoring data, real-time reporting, etc.).
  • Familiarity with call centre data and technologies for real-time transcriptions and natural language processing is a nice to have.
  • Ability to collaborate with Data Science teams to solve complex challenges, including exploring Generative AI applications.
  • Experience deploying ML models in practice is desirable, particularly around the integration of models into real-time data workflows.
  • Experience leading or mentoring others, ensuring high performance and fostering a collaborative culture.
  • A passion for tackling complex data challenges and being part of a transformative journey towards real-time, AI-driven solutions.
  • Experience in the insurance industry or similar sectors is a plus but not essential.

  
Why should the Principal / Lead Data Engineer join? :

  • A great opportunity to own and shape the future of data engineering within a global organization, driving key innovations in real-time data solutions.
  • Work on exciting real-time data use cases, including scoring, transcription, and AI model training for large-scale operations like contact centres.
  • Play a key role in the development and deployment of Generative AI technologies, with a focus on practical, impactful use cases.
  • Enjoy the flexibility of a hybrid work environment, balancing work and life commitments.
  • Competitive salary of up to $220k base plus superannuation.

  
This is a unique opportunity for a proactive, hands-on Data Engineering Lead to take ownership of critical data initiatives, solve complex challenges, and work alongside a talented team on innovative real-time data and AI applications. If you're ready to shape the future of data engineering and play a pivotal role in driving transformative solutions, apply now
  
Keywords: Microsoft Azure / Power BI / Azure Data Factory / Databricks / SQL / Python / Insurance / Principal Data Engineer / Lead Data Engineer

  • Principal Data Engineer

    1 month ago

    Sydney, Australia Insignia Financial Full time

    Join a motivated and talented Data team Play a key technical lead role in the transformation that will be utilised by the entire business Lead and mentor a team to develop their skills and deliver cutting-edge solutions The opportunity to join our teamWe have an exciting opportunity for a Principal Data Engineer – Google Cloud Platform (GCP) to...

  • Principal / Senior Data Engineer

    2 months ago

    Sydney, Australia Simple Machines Full time

    Simple Machines ANZ – Job Ad – Principal / Senior Data EngineerPosition: Senior Data EngineerLocation: Darlinghurst, SydneySimple Machines. Data Engineered to Life™ Simple Machines is a leading independent boutique technology firm with a global presence, including teams in London, Sydney, San Francisco, and New Zealand. We specialise in creating...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Data - AI & Data Science Full time

    Drive Business Value with Data Engineering at Domain GroupWe are seeking a skilled Data Engineer to join our Data Decision Science team in Sydney. As a key member of our data department, you will play a critical role in building scalable data pipelines that power Australia's property market.About the RoleThis is an exciting opportunity to work with...

  • Principal Data Engineer

    3 weeks ago

    Sydney, Australia Neilson Financial Services Full time

    Job DescriptionKey Responsibilities:Design and implement the enterprise-wide data architecture strategy, including hands-on development of proof-of-concepts for new technologies and frameworksArchitect and actively develop complex distributed systems that can scale across multiple regions and handle diverse data workloads Develop, manage, and optimize ETL...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Data - AI & Data Science Full time

    About UsWe shine a light on the property market, simplifying the journey for all involved. Our expertise and exclusive data drive change. At Domain Group, our data department leads the way in driving value across the organisation.The RoleAs a Senior Data Engineer, you'll play a key role in building data pipelines that power Australia's property market....

  • Senior Data Engineer- SVP

    4 weeks ago

    Sydney, Australia Data-Property Data Solns Shared Full time

    Who are weWe shine a light on all things property. Our business aims to simplify the property journey for all involved; motivated by expertise and our exclusive data. Changing the way people engage with property requires a team of diverse thinkers.Our data department is at the forefront of driving value across the organisation. We create reusable data assets...

  • Principal Data Platform Engineeer

    6 months ago

    Sydney, Australia Atlassian Full time

    Overview: **The Role**: We're hiring a Principal Data Platform Engineer reporting to the Engineering Manager for the Internal Analytics Platform. You are an expert Cloud Engineer/SRE with expertise across a range of infrastructure technologies e,g AWS, Kubernetes. You will join the team that oversees Socrates, Atlassian's internal data lake. The team is...

  • Data Property Solutions Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Data-Property Data Solns Shared Full time

    We are seeking a highly skilled Solutions Data Engineer to join our team at Data-Property Data Solns Shared. As a Senior member of our organization, you will play a critical role in shaping the future of property data solutions.About UsWe shine a light on all things property, aiming to simplify the property journey for all involved. Our business is motivated...

  • Principal Engineer

    7 months ago

    Sydney, Australia Amazon Full time

    DESCRIPTIONPrincipal Engineers provide technical leadership at Amazon.com. They help establish technical standards and drive Amazon’s overall technical architecture, engineering practices, and engineering methodologies. They work on our hardest problems, building high quality, architecturally sound systems that are aligned with our business needs. They...

  • Data Engineer for Property Insights

    4 weeks ago

    Sydney, New South Wales, Australia Data-Property Data Solns Shared Full time

    We are seeking a highly skilled Senior Data Engineer to join our Single View Property (SVP) team at Domain. This role is perfect for someone who thrives in a dynamic environment and has a passion for working with data to drive business outcomes.

  • Principal Engineer

    7 months ago

    North Sydney, Australia Amazon Web Services Australia Pty Ltd - D81 Full time

    Principal Engineers provide technical leadership at Amazon.com. They help establish technical standards and drive Amazon’s overall technical architecture, engineering practices, and engineering methodologies. They work on our hardest problems, building high quality, architecturally sound systems that are aligned with our business needs. They think globally...

  • Senior Data Engineer- Data Decision Science Group

    4 weeks ago

    Sydney, Australia Data - AI & Data Science Full time

    Who are weWe shine a light on all things property. Our business aims to simplify the property journey for all involved; motivated by expertise and our exclusive data. Changing the way people engage with property requires a team of diverse thinkers.At Domain Group, our data department is at the forefront of driving value across the organisation. We create...

  • Principal Engineer

    1 month ago

    Sydney, Australia Qantas Full time

    Job DescriptionThe Principal Engineer is a technical leadership role, responsible for driving technical outcomes of the team and advising and supporting best across the business within flight operations, airline and mobile technologies.  The Principal Software Engineer will have mastered the fundamentals of software development, from problem exploration...

  • Principal Engineer

    1 month ago

    Sydney, Australia Qantas Airways Limited Full time

    Job DescriptionThe Principal Engineer is a technical leadership role, responsible for driving technical outcomes of the team and advising and supporting best across the business within flight operations, airline and mobile technologies.  The Principal Software Engineer will have mastered the fundamentals of software development, from problem exploration...

  • Principal Engineer

    3 weeks ago

    Sydney, Australia Qantas Full time

    Job DescriptionThe Principal Engineer is a technical leadership role, responsible for driving technical outcomes of the team and advising and supporting best across the business within flight operations, airline and mobile technologies.  The Principal Software Engineer will have mastered the fundamentals of software development, from problem exploration...

  • Senior Data Engineer

    3 weeks ago

    Sydney, New South Wales, Australia Data-Property Data Solns Shared Full time

    We are seeking a highly skilled Senior Data Engineer to join our team at Data-Property Data Solutions Shared. This is an exciting opportunity for a seasoned professional to lead the development of cutting-edge data solutions and drive business growth.About UsOur company specializes in simplifying the property journey for all stakeholders, leveraging our...

  • Data Engineering Expert

    4 weeks ago

    Sydney, New South Wales, Australia Data - AI & Data Science Full time

    About UsWe shine a light on all things property, simplifying the journey for everyone involved. Our expertise and exclusive data drive value across the organisation.Role OverviewThis is an exciting opportunity to make your mark in data engineering at Domain Group's Data Decision Science team. You'll play a key role in building data pipelines that power...

  • Principal Data Consultant

    6 months ago

    Sydney, Australia DiUS Computing Full time

    **Location: This role is open in Melbourne or Sydney** **About The Role** Our clients are varied, and we are on the lookout for an experienced data expert with a commercial data engineering background, who can confidently navigate a client on what they can do with their data. You will be drawing from past experiences in a consulting firm, where you have...

  • Senior Azure Data Engineer

    8 months ago

    North Sydney, Australia NTT DATA Full time

    NTT DATA is heavily investing globally in providing state of the art analytical solutions to its clients. Our capability has been focused locally, regionally in APAC and globally by forming strategic partnerships including Microsoft Azure, SAP Analytics, Snowflake, Matillion and many more, leveraging the strength of our cloud and network assets via the NTT...

  • Principal Engineer

    2 months ago

    Sydney, Australia VGW Full time

    Principal Engineer VGW is an interactive entertainment company, harnessing technology and creativity to deliver world-class, free-to-play games.  We have an exciting opportunity to join our Engineering team in Sydney and are currently looking for a Principal Engineer to join the team. Join us as a Principal Engineer and shape the architecture of high-load,...