Data Engineering Lead

2 weeks ago

Sydney, New South Wales, Australia Talent Insights Group Full time
Senior Data Consultant at Talent Insights

Location: Sydney (Hybrid Work)

Position Type: Full-Time, Permanent

An exciting opportunity has arisen for a hands-on, proactive Data Engineering Lead to join a leading Insurance company based in Sydney.

In this role, you will be responsible for leading a team of 2-4 data engineers, working as part of a global team on high-impact data engineering initiatives. This is an excellent opportunity to own and shape the data engineering function, drive innovation, and play a key role in building real-time data solutions using Microsoft Azure and Generative AI technologies.

Key Responsibilities for the Data Engineering Lead include:

  • Lead a team of 2-4 data engineers, guiding and mentoring them to deliver high-quality data engineering solutions in a fast-paced environment.
  • Take ownership of building, optimizing, and scaling real-time data pipelines using Microsoft Azure, including Azure Databricks, Azure Data Factory, and Power BI.
  • Work with real-time data use cases: scoring data in real-time, ensuring that information is transferred and processed quickly, with an emphasis on low-latency data flow.
  • Play a key role in the development and maintenance of war boards in contact centres worldwide, enabling real-time dashboards and reporting with near-zero delays.
  • Focus on solving complex data challenges related to processing thousands of calls coming through call centers, including real-time transcriptions and passing the data for further processing.
  • Partner with the Data Science team and be involved in a journey of real-time training and AI model development, including working on training your own ChatGPT model for specific business use cases.
  • Explore and implement Generative AI applications within the data engineering function, particularly around real-time transcriptions and training models that will drive future capabilities.
  • Collaborate with the broader technology team to continuously improve data quality, security, and pipeline performance.
  • Be proactive in problem-solving, owning data engineering challenges and delivering impactful solutions under tight deadlines.

Key Skills & Experience required for the Data Engineering Lead include:

  • Hands-on experience with Microsoft Azure, ideally including Azure Databricks, Azure Data Factory, and Power BI.
  • Strong proficiency in Python and SQL for building scalable and efficient data pipelines.
  • Experience in real-time data processing, with a focus on minimizing delays (e.g., 1-second latency in scoring data, real-time reporting, etc.).
  • Familiarity with call centre data and technologies for real-time transcriptions and natural language processing is a nice to have.
  • Ability to collaborate with Data Science teams to solve complex challenges, including exploring Generative AI applications.
  • Experience deploying ML models in practice is desirable, particularly around the integration of models into real-time data workflows.
  • Experience leading or mentoring others, ensuring high performance and fostering a collaborative culture.
  • A passion for tackling complex data challenges and being part of a transformative journey towards real-time, AI-driven solutions.
  • Experience in the insurance industry or similar sectors is a plus but not essential.

Why should the Data Engineering Lead join?

  • A great opportunity to own and shape the future of data engineering within a global organization, driving key innovations in real-time data solutions.
  • Work on exciting real-time data use cases, including scoring, transcription, and AI model training for large-scale operations like contact centres.
  • Play a key role in the development and deployment of Generative AI technologies, with a focus on practical, impactful use cases.
  • Enjoy the flexibility of a hybrid work environment, balancing work and life commitments.
  • Competitive salary of up to $220k base plus superannuation.

This is a unique opportunity for a proactive, hands-on Data Engineering Lead to take ownership of critical data initiatives, solve complex challenges, and work alongside a talented team on innovative real-time data and AI applications. If you're ready to shape the future of data engineering and play a pivotal role in driving transformative solutions, apply now

Seniority level

Director

Employment type

Full-time

Job function

Information Technology and Engineering

Industries

Insurance and Information Services

#J-18808-Ljbffr

  • Data Engineer

    5 days ago

    Sydney, New South Wales, Australia rich Data Co Full time

    About RDCRich Data Co (RDC) Delivering the Future of Credit, Today We believe credit should be accessible, fair, inclusive, and sustainable. We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way. Global leading financial institutions are leveraging...

  • Rich Data Co. Data Engineering Professional

    5 days ago

    Sydney, New South Wales, Australia rich Data Co Full time

    Join the Future of Credit5-Star Employer of Choice 2023: RDC was named one of HRD Australia's best companies to work for in Australia.Work at a fast-growing global AI company and grow your skills, capabilities, and gain AI and global experience.High performance team: Work alongside some of the best product teams, data scientists, and credit experts in the...

  • Data Engineering Lead

    3 days ago

    Sydney, New South Wales, Australia Latitude IT Full time

    We are seeking a highly experienced Data Engineering Tech Lead to drive the strategic and technical development of an enterprise-wide Azure Data Platform.About the RoleThis role involves leading the technical direction for the Azure Data Platform, designing, developing, and implementing scalable, reusable data pipelines.Provide technical leadership and...

  • Lead Data Engineer

    2 days ago

    Sydney, New South Wales, Australia Pyramid Global Technologies Full time

    Job Description for Lead Data Engineer in SydneyExperience: 10+ years in Software Engineering, Cloud Computing, Databricks, and Big Data.Key Responsibilities:Data Engineering and Data WarehousingSolution Design and Data AnalysisData Visualization and Data ModelingProgramming Languages: Python, Scala, SQLCloud Technologies: Azure and DatabricksBig Data...

  • AI and Data Engineering Specialist

    5 days ago

    Sydney, New South Wales, Australia rich Data Co Full time

    Become Part of Our TeamYou will be responsible for designing and building real-time data integration solutions that ensure timely and accurate reporting. Collaborate with stakeholders to gather data requirements and refine report outputs based on feedback. Stay updated on the latest trends in data engineering and dashboard development, focusing on optimizing...

  • Lead Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Lead Data EngineerSydney, Australia$230,000 (including Super) + BonusAbout the Role:Join a high-performing Data Team at a leading financial services institution to lead regulatory initiatives and drive innovation in data engineering, utilizing AI and automation to enhance platform efficiency.Key Responsibilities:Lead the development and transformation of the...

  • Lead Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Pyramid Global Technologies Full time

    Job Description for Lead Data Engineer in Sydney.Experience: 10+ years in Software Engineering, Cloud Computing, Databricks, and Big Data.Key Responsibilities:Data EngineeringData WarehousingSolution DesignData AnalysisVisualizationData ModelingProgramming Languages: Python, Scala, SQLCloud Technologies: Azure and DatabricksBig Data Technologies: Cloudera...

  • Lead Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Paxus - Technology + Digital Talent Full time

    Permanent Opportunity open to interstate candidatesGenerous remunerationFlexible Working ArrangementOur client is looking for a Lead Data Engineer to shape their data platform and strategy. You will be the leader of making technical decisions, ensuring operational excellence, best engineering practices, and strategic vision across multiple projects.Key...

  • Data Engineering Lead

    3 days ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Job Summary:We are seeking an experienced Data Engineering Lead to join our team at ClearCompany.This is a 12 + 12 month contract opportunity for two lead data engineers who will be responsible for designing, building, testing, and maintaining data pipelines to move data between systems.The ideal candidate will have five years of experience working as a Data...

  • Data Engineering Lead

    1 week ago

    Sydney, New South Wales, Australia Macquarie Bank Limited Full time

    Join our Group Treasury Transformation Liquidity Technology team as a Lead Data Engineer. We carry an exciting program of work to deliver our next-generation Liquidity Platform. This is an excellent opportunity for someone with a keen interest in Finance (Group Treasury), Data, and Technology.At Macquarie, our advantage is bringing together diverse people...

  • Lead Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Macquarie Bank Limited Full time

    Join our Customer Intelligence Engine (CIE) Technology team and help us build the best technology business in financial services. This is a great opportunity to be in a leadership as well as hands-on role while working with a team of talented engineers in developing the next generation Customer Intelligence Engine.At Macquarie, our advantage is bringing...

  • Lead Data Engineer

    3 days ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Design, build, test and maintain data pipelines to move data between systemsNV1 clearance required - 12 + 12 month contractSurry Hills, NSW - Flexible work arrangementsOur client, a large Federal government agency based in Surry Hills, NSW, requires two lead data engineers for 12 + 12 month contracts.DutiesDesign: Creating scalable data pipelines that meet...

  • Big Data Engineer Lead

    2 days ago

    Sydney, New South Wales, Australia Ampstek Full time

    About the RoleWe are seeking a Big Data Engineer Lead to join our team and lead the development of our data infrastructure. The successful candidate will be responsible for designing and implementing efficient ETL processes using Databricks, ADF, and SQL. This is an excellent opportunity for someone who is passionate about working with big data and wants to...

  • Data Engineering Lead

    2 days ago

    Sydney, New South Wales, Australia Expert360 Full time

    About Expert360Expert360 is a platform that empowers elite professionals to find great work and be successful. Our mission is to connect amazing people with great companies to get important work done.We believe in choice, flexibility, and empowerment. We believe if you can change the world of work, you can change the world. At Expert360, we're defining and...

  • Data Engineering Lead Position

    7 days ago

    Sydney, New South Wales, Australia PayTech Group Full time

    About the RoleThis is an exciting opportunity for a Senior Data Engineer to join our team at PayTech Group and take on a leadership role in shaping our Big Data and Data Warehouse strategy.The successful candidate will have experience in designing, building, and delivering optimized enterprise-wide data ingestion, data integration, and data pipeline...

  • Data Engineering Lead

    1 day ago

    Sydney, New South Wales, Australia Pyramid Global Technologies Full time

    Company Overview:Pyramid Global Technologies is a leading provider of innovative technology solutions.We are seeking an experienced Data Engineering Lead to join our team in Sydney.

  • Senior Data Engineering Lead

    6 days ago

    Sydney, New South Wales, Australia Westpac Group Full time

    We are seeking a highly skilled Senior Data Engineering Lead to join our team at Westpac Group. As a leader in the field, you will be responsible for driving the delivery of next-generation technology capabilities, including data analytics, platform development, and software development.">Key Responsibilities:Provide technical expertise and lead dedicated...

  • Senior Data Engineer Lead

    3 days ago

    Sydney, New South Wales, Australia Qantas Airways Full time

    Job Summary:We are seeking a skilled Senior Data Engineer to join our Data & Analytics team at Qantas Airways. As a key member of our team, you will play a vital role in driving innovation and providing technical direction input to Architect/Delivery Lead/Product Owner.Key Responsibilities:Collaborate closely with Data Modellers, Technical Business Analysts,...

  • Cloud Data Engineer Lead

    1 day ago

    Sydney, New South Wales, Australia Commonwealth Bank Full time

    We are looking for an experienced Cloud Data Engineer Lead to oversee the development and deployment of cloud-based data engineering solutions.As part of our CDAO team, you will be responsible for leading cross-functional teams in delivering high-quality data products that meet business requirements.About the Role:You will be responsible for designing,...

  • Lead Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Mantel Group Pty Ltd Full time

    About usMantel Group is an Australian-owned technology consulting business with capabilities across Cloud, Digital, Data, Delivery & Security. Since our inception in November 2017, we have experienced remarkable growth across Australia & New Zealand and are honoured to be recognised as a Great Place to Work for 5 years in a rowWe hire smart and talented...