Data Engineering Lead

2 weeks ago


Sydney, New South Wales, Australia Talent Insights Group Full time
Senior Data Consultant at Talent Insights

Location: Sydney (Hybrid Work)

Position Type: Full-Time, Permanent

An exciting opportunity has arisen for a hands-on, proactive Data Engineering Lead to join a leading Insurance company based in Sydney.

In this role, you will be responsible for leading a team of 2-4 data engineers, working as part of a global team on high-impact data engineering initiatives. This is an excellent opportunity to own and shape the data engineering function, drive innovation, and play a key role in building real-time data solutions using Microsoft Azure and Generative AI technologies.

Key Responsibilities for the Data Engineering Lead include:

  • Lead a team of 2-4 data engineers, guiding and mentoring them to deliver high-quality data engineering solutions in a fast-paced environment.
  • Take ownership of building, optimizing, and scaling real-time data pipelines using Microsoft Azure, including Azure Databricks, Azure Data Factory, and Power BI.
  • Work with real-time data use cases: scoring data in real-time, ensuring that information is transferred and processed quickly, with an emphasis on low-latency data flow.
  • Play a key role in the development and maintenance of war boards in contact centres worldwide, enabling real-time dashboards and reporting with near-zero delays.
  • Focus on solving complex data challenges related to processing thousands of calls coming through call centers, including real-time transcriptions and passing the data for further processing.
  • Partner with the Data Science team and be involved in a journey of real-time training and AI model development, including working on training your own ChatGPT model for specific business use cases.
  • Explore and implement Generative AI applications within the data engineering function, particularly around real-time transcriptions and training models that will drive future capabilities.
  • Collaborate with the broader technology team to continuously improve data quality, security, and pipeline performance.
  • Be proactive in problem-solving, owning data engineering challenges and delivering impactful solutions under tight deadlines.

Key Skills & Experience required for the Data Engineering Lead include:

  • Hands-on experience with Microsoft Azure, ideally including Azure Databricks, Azure Data Factory, and Power BI.
  • Strong proficiency in Python and SQL for building scalable and efficient data pipelines.
  • Experience in real-time data processing, with a focus on minimizing delays (e.g., 1-second latency in scoring data, real-time reporting, etc.).
  • Familiarity with call centre data and technologies for real-time transcriptions and natural language processing is a nice to have.
  • Ability to collaborate with Data Science teams to solve complex challenges, including exploring Generative AI applications.
  • Experience deploying ML models in practice is desirable, particularly around the integration of models into real-time data workflows.
  • Experience leading or mentoring others, ensuring high performance and fostering a collaborative culture.
  • A passion for tackling complex data challenges and being part of a transformative journey towards real-time, AI-driven solutions.
  • Experience in the insurance industry or similar sectors is a plus but not essential.

Why should the Data Engineering Lead join?

  • A great opportunity to own and shape the future of data engineering within a global organization, driving key innovations in real-time data solutions.
  • Work on exciting real-time data use cases, including scoring, transcription, and AI model training for large-scale operations like contact centres.
  • Play a key role in the development and deployment of Generative AI technologies, with a focus on practical, impactful use cases.
  • Enjoy the flexibility of a hybrid work environment, balancing work and life commitments.
  • Competitive salary of up to $220k base plus superannuation.

This is a unique opportunity for a proactive, hands-on Data Engineering Lead to take ownership of critical data initiatives, solve complex challenges, and work alongside a talented team on innovative real-time data and AI applications. If you're ready to shape the future of data engineering and play a pivotal role in driving transformative solutions, apply now

Seniority level

Director

Employment type

Full-time

Job function

Information Technology and Engineering

Industries

Insurance and Information Services

#J-18808-Ljbffr
  • Data Engineer

    2 weeks ago


    Sydney, New South Wales, Australia NTT DATA, Inc. Full time

    Get AI-powered advice on this job and more exclusive features.Direct message the job poster from NTT DATA, Inc.Location: Sydney Olympic Park (Hybrid or Onsite Preferred)Citizenship is mandatory.Remote Work: Available for candidates with NV1 Security ClearanceDuration: 2–3 Weeks (with potential for extension)Start Date: ImmediateAbout the Role:We are...

  • Data Engineer

    2 weeks ago


    Sydney, New South Wales, Australia NTT DATA, Inc. Full time

    Get AI-powered advice on this job and more exclusive features.Direct message the job poster from NTT DATA, Inc.Location: Sydney Olympic Park (Hybrid or Onsite Preferred)Citizenship is mandatory.Remote Work: Available for candidates with NV1 Security ClearanceDuration: 2–3 Weeks (with potential for extension)Start Date: ImmediateAbout the Role:We are...


  • Sydney, New South Wales, Australia Rich Data Co Full time

    About RDCRich Data Co (RDC) Delivering the Future of Credit, Today We believe credit should be accessible, fair, inclusive, and sustainable. We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way. Global leading financial institutions are leveraging...

  • Data Engineering Lead

    4 weeks ago


    Sydney, New South Wales, Australia Talent Insights Group Full time

    Senior Data Consultant at Talent InsightsLocation: Sydney (Hybrid Work)Position Type: Full-Time, PermanentAn exciting opportunity has arisen for a hands-on, proactive Data Engineering Lead to join a leading Insurance company based in Sydney.In this role, you will be responsible for leading a team of 2-4 data engineers, working as part of a global team on...


  • Sydney, New South Wales, Australia beBeeEngineering Full time

    Job Title: Data Engineering LeadWe are seeking a highly skilled and experienced Data Engineering Lead to join our team. This is an exciting opportunity for someone who wants to make a difference by helping shape a healthier, better-protected Australia.The successful candidate will lead detailed technical design and do hands-on development along with...


  • Sydney, New South Wales, Australia beBeeDataEngineering Full time

    Job TitleData Engineering LeadThis is an exciting opportunity to join a great culture and environment with a company while they continue to scale and grow with exciting new projects. As the Data Engineering Lead, you will drive data engineering best practices, architecture, and develop roadmaps in order to deliver quality data throughout the organization.Key...

  • Lead Data Engineer

    1 week ago


    Sydney, New South Wales, Australia beBee Careers Full time

    Lead Data EngineerWe're seeking a skilled Lead Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems.About the RoleDesign and develop scalable data pipelines using industry-standard tools and technologies.Collaborate with...

  • Lead Data Engineer

    3 weeks ago


    Sydney, New South Wales, Australia Pyramid Global Technologies Full time

    Job Description for Lead Data Engineer in SydneyExperience: 10+ years in Software Engineering, Cloud Computing, Databricks, and Big Data.Key Responsibilities:Data Engineering and Data WarehousingSolution Design and Data AnalysisData Visualization and Data ModelingProgramming Languages: Python, Scala, SQLCloud Technologies: Azure and DatabricksBig Data...


  • Sydney, New South Wales, Australia beBee Careers Full time

    Data Engineering Lead - Transformative Role in Real-Time Data and AI ApplicationsWe are seeking a proactive and experienced leader to lead our data engineering efforts, responsible for guiding and mentoring a team of data engineers to deliver high-quality solutions.Take ownership of building, optimizing, and scaling real-time data pipelines using Microsoft...

  • Data Engineer Lead

    1 week ago


    Sydney, New South Wales, Australia beBee Careers Full time

    Job DescriptionWe are seeking a dynamic and experienced Lead Engineer to join our innovative team and drive the future of healthcare technology.About The RoleThe primary role of the Analytics Engineering function within the Global Data Platform, Global Technology Solutions team is to design and deliver new (or improved) data engineering solutions (data...