Data Engineer

3 weeks ago

Sydney, New South Wales, Australia Talent Insights Group Full time

Get AI-powered advice on this job and more exclusive features.

Senior Data Consultant at Talent Insights

Data Engineer (Databricks / Airflow / Azure DF)

Location: Sydney Hybrid

Duration: 6 months initial contract, likely to extend

Rates: A$800-1000 a day inc super

Talent Insights is currently partnered with a leading Financial institution committed to driving innovation and delivering high-quality data solutions. We are looking for an experienced Data Engineer with expertise in Databricks migrations to join for a 6-month contract role. This is an excellent opportunity for someone with strong technical skills and a passion for working with cutting-edge data technologies to work on a greenfield project and to have complete autonomy over a migration to Databricks.

Role Overview:

As a Data Engineer, you will play a key role in building and maintaining data pipelines and infrastructure that support the organization's data-driven decision-making processes. You will work closely with data science, analytics, and product teams to implement robust, scalable solutions. The ideal candidate will have hands-on experience with Databricks migrations and the ability to work in a fast-paced, collaborative environment.

Key Responsibilities:

  • Design, build, and maintain scalable data pipelines using Databricks.
  • Develop ETL processes for ingesting, transforming, and storing large datasets.
  • Collaborate with data scientists and analysts to ensure efficient data processing and accessibility.
  • Optimize the performance of data pipelines to handle large volumes of data efficiently.
  • Ensure data quality, consistency, and reliability across various datasets.
  • Work with cloud-based platforms (e.g., AWS, Azure) and integrate with Databricks.
  • Implement best practices for data governance, security, and compliance.
  • Troubleshoot and resolve issues related to data pipeline performance and reliability.
  • Document processes, procedures, and architecture to ensure smooth handovers and knowledge transfer.

Required Skills & Experience:

  • Proven experience as a Data Engineer with hands-on expertise in Databricks.
  • Strong proficiency in SQL, Python, and/or Scala for data processing and automation.
  • Experience working with cloud platforms such as AWS or Azure.
  • Familiarity with data lake architecture and large-scale distributed data systems.
  • Understanding of ETL processes, data modeling, and data integration.
  • Ability to optimize and maintain data pipelines for performance and scalability.
  • Experience working with data storage solutions like Delta Lake, Apache Spark, and other big data tools.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration skills to work across teams.

Preferred Skills:

  • Experience with data orchestration tools like Apache Airflow or similar.
  • Knowledge of machine learning workflows or AI tools integrated with Databricks.
  • Familiarity with DevOps practices and CI/CD pipelines for data infrastructure.

Seniority level: Mid-Senior level

Employment type: Contract

Job function: Engineering and Information Technology

Industries: Financial Services, Technology, Information and Media, and IT Services and IT Consulting

#J-18808-Ljbffr

  • Data Engineer

    3 weeks ago

    Sydney, New South Wales, Australia rich Data Co Full time

    About RDCRich Data Co (RDC) Delivering the Future of Credit, Today We believe credit should be accessible, fair, inclusive, and sustainable. We are passionate about AI and developing new techniques that leverage traditional and non-traditional data to get the right decision in a clear and explainable way. Global leading financial institutions are leveraging...

  • Cloud Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia One51 | Data & Analytics Consultancy Full time

    We are seeking an experienced Cloud Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing and developing scalable data architectures using Microsoft Azure.About the Role:Design and implement ETL pipelines using Azure Data Factory (ADF) and ELT processes.Develop and maintain large-scale data...

  • Data Analyst for Search Engine Improvement

    2 weeks ago

    Sydney, New South Wales, Australia Welo Data Full time

    Job DescriptionWe are seeking a highly skilled Data Analyst to join our team in improving search engine quality. As a Web Information Assessor, you will play a crucial role in ensuring the accuracy and relevance of search results.About the JobReview and assess internet search results to evaluate their relevance and accuracy.Provide feedback to improve search...

  • Microsoft Azure Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia One51 | Data & Analytics Consultancy Full time

    We are looking for a Microsoft Azure Data Engineer who will be responsible for developing and maintaining the data architecture, data models, and standards for various Data Integration & Data Warehouse projects in Azure. Ensure new features and subject areas are modeled to integrate with existing structures and provide a consistent view. In addition, develop...

  • Data Solutions Architect

    2 weeks ago

    Sydney, New South Wales, Australia Data Addiction Full time

    About UsData Addiction is a leading provider of Microsoft solutions, serving clients across ANZ with expertise in data, AI, D365, and Azure. Our mission is to deliver best-in-class solutions that drive business value.Job OverviewWe are seeking a Senior Data Specialist to join our team in Sydney or Melbourne. As a key member of our team, you will be...

  • Cloud Data Engineering Manager

    2 weeks ago

    Sydney, New South Wales, Australia Data Addiction Full time

    Our TeamAt Data Addiction, we offer a strong learning and development environment, hybrid work-from-home/office arrangements, and an extra annual leave day for every year of service. We also provide career development funding towards your professional growth with clear development frameworks, a home office allowance, wellness days, and a commitment to...

  • Senior Data Solutions Architect

    2 weeks ago

    Sydney, New South Wales, Australia Data Addiction Full time

    About the JobThe successful candidate will have 5+ years of experience building cloud-based data solutions, preferably in Microsoft Azure. A proven track record in leading projects using Agile/Scrum methods is essential. Strong problem-solving skills and excellent verbal and written communication skills are also required.Tertiary qualifications in Computer...

  • Data Engineer

    1 day ago

    Sydney, New South Wales, Australia Mutinex Full time

    Data Engineer Department: EngineeringEmployment Type: Full TimeLocation: SydneyReporting To: Dan IvanovCompensation: $110,000 - $130,000 / year Description We're an early-stage B2 B Saa S startup with a proven platform, big-name clients, and millions in revenue.We're not chasing unicorn status; we're building a sustainable, long-lasting business (think...

  • Microsoft Senior Data Specialist

    4 weeks ago

    Sydney, New South Wales, Australia Data Addiction Full time

    Direct message the job poster from Data AddictionOur Microsoft practice with Seisma Group (formerly Data Addiction), continues to grow across ANZ. We are leaders in delivering Microsoft solutions for clients across Data, AI, D365 and Azure. If you are a data person at heart, you love Microsoft and you want to grow rapidly in your career, this is the place...

  • Microsoft Senior Data Specialist

    2 weeks ago

    Sydney, New South Wales, Australia Data Addiction Full time

    Direct message the job poster from Data AddictionOur Microsoft practice with Seisma Group (formerly Data Addiction), continues to grow across ANZ. We are leaders in delivering Microsoft solutions for clients across Data, AI, D365 and Azure. If you are a data person at heart, you love Microsoft and you want to grow rapidly in your career, this is the place...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Mutinex Full time

    Data EngineerDepartment: EngineeringEmployment Type: Full TimeLocation: SydneyReporting To: Dan IvanovCompensation: $110,000 - $130,000 / yearDescription We're an early-stage B2B SaaS startup with a proven platform, big-name clients, and millions in revenue. We're not chasing unicorn status; we're building a sustainable, long-lasting business (think Bowhead...

  • Data Engineer

    2 days ago

    Sydney, New South Wales, Australia Mutinex Full time

    Data EngineerDepartment: EngineeringEmployment Type: Full TimeLocation: SydneyReporting To: Dan IvanovCompensation: $110,000 - $130,000 / yearDescription We're an early-stage B2B SaaS startup with a proven platform, big-name clients, and millions in revenue. We're not chasing unicorn status; we're building a sustainable, long-lasting business (think Bowhead...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Optiver Full time

    This is a Data Engineer role with one of the leading companies in AU right now -- Optiver -- with an amazing team. They are continuing to grow rapidly. This is the chance to join right as the takes off.More About the Role at Optiver **WHAT YOU'LL DO** Optiver Sydney is seeking a seasoned Data Engineer to support and advance the data needs of our local...

  • Data Engineer

    3 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    12 + 12 + 12 month contract - ASAP Start$130 - $160 per hour (inc-Super) - SFIA4/5Must hold TSPV Security Clearance - Multiple positions availableCanberra or Sydney Address - 100% onsiteOur client, a very large Federal Government Organisation, whose focus is on national security, emergency management, border control, immigration and multicultural affairs, is...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Maltem Australia Full time

    Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Maltem AustraliaWe support organisations in building their enterprise data platform end-to-endAbout Us:Maltem Australia is a global boutique data and analytics consulting firm specialising in supporting organisations in designing and delivering enterprise data...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Kindred Group plc Full time

    The RoleOur Sportsbook platform is to be the most insight driven betting platform and in the last few years we've invested heavily in our data capabilities. Focussing on our Sportsbook product, our Data Engineers will play a critical role in streamlining and optimising data management processes, ensuring faster, more reliable, and data-driven...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Kindred Group plc Full time

    The RoleOur Sportsbook platform is to be the most insight driven betting platform and in the last few years we've invested heavily in our data capabilities. Focussing on our Sportsbook product, our Data Engineers will play a critical role in streamlining and optimising data management processes, ensuring faster, more reliable, and data-driven...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Expert360 Full time

    Direct message the job poster from Expert360Connecting Experts to Great OpportunitiesTitle: Consultant - Data EngineerLocation: SydneyDuration: 6 months initial term with the possibility of extensionEmployment Type: ContractAbout Expert360:Expert360 empowers more than 42,000 (and growing) elite professionals around the world to find great work and be...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Optiver US LLC Full time

    Spot a fraudulent website or app impersonating Optiver? Report it here. This message will disappear as you scroll or click the close icon to hide it permanently.WHO WE AREOptiver is a global market maker founded in Amsterdam, with offices in London, Chicago, Austin, New York, Sydney, Shanghai, Hong Kong, Singapore, Taipei and Mumbai. Established in 1986,...

  • Data Engineer

    3 weeks ago

    Sydney, New South Wales, Australia Optiver US LLC Full time

    Spot a fraudulent website or app impersonating Optiver? Report it here. This message will disappear as you scroll or click the close icon to hide it permanently.WHO WE AREOptiver is a global market maker founded in Amsterdam, with offices in London, Chicago, Austin, New York, Sydney, Shanghai, Hong Kong, Singapore, Taipei and Mumbai. Established in 1986,...