Data Engineer

1 week ago

Sydney, New South Wales, Australia Talent International Full time

Our client is seeking a skilled and motivated Data Engineer to join their dynamic team. The ideal candidate will possess extensive experience with Microsoft Azure, Databricks, real-time integrations, and data streaming (Kafka). You will be responsible for designing, building, and maintaining their data infrastructure to support their data analytics and business intelligence needs. Your expertise in data lakes, SQL procedures, and CICD pipelines will be critical to ensuring efficient and reliable data processes.

Responsibilities:

  1. Azure Data Solutions:
    • Design, implement, and manage data solutions using Azure Blob Storage, Azure Kubernetes Service (AKS), and Azure Data Factory (ADF).
    • Ensure the scalability and reliability of our Azure-based data infrastructure.
  2. Databricks Development:
    • Develop and maintain data pipelines using Databricks, with a focus on PySpark and Python.
    • Optimize data workflows for performance and cost-efficiency within the Databricks environment.
  3. Real-time Data Integration:
    • Design and implement real-time data integration solutions using data streaming technologies such as Kafka, Azure Functions, ADF, and Flink.
    • Develop and maintain CICD pipelines to automate deployment and monitoring of data streaming processes.
  4. Data Lake Construction:
    • Build and maintain data lakes on Databricks to support scalable and flexible data storage and analytics.
    • Ensure data quality, consistency, and security within the data lake environment.
  5. SQL Development:
    • Write, debug, and optimize complex PL/SQL and T-SQL procedures.
    • Collaborate with data analysts and other stakeholders to meet their data querying and reporting needs.

Requirements:

  • 6-10 years of experience in data engineering or a related role.
  • Strong expertise in Microsoft Azure services, including Azure Blob Storage, AKS, and ADF.
  • Proficiency in Databricks with a focus on PySpark and Python.
  • Hands-on experience with real-time data integration and streaming technologies (Kafka, Azure Functions, ADF, Flink).
  • Proven experience building and maintaining data lakes on Databricks.
  • Strong knowledge of PL/SQL and T-SQL, with hands-on experience in writing and debugging SQL procedures.
  • Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
  • Strong communication skills and the ability to work effectively with cross-functional teams.

Preferred Qualifications:

  • Master's degree in Computer Science, Information Technology, or a related field.
  • Certifications in Microsoft Azure and/or Databricks.
  • Experience with other data integration and ETL tools.
  • Familiarity with additional programming languages and data processing frameworks.

If you think you have above skills and experiences, click 'Apply' button or send your resume to

  • Senior Azure Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia NTT DATA Full time

    Job Description NTT DATA is heavily investing globally in providing state of the art analytical solutions to its clients. Our capability has been focused locally, regionally in APAC and globally by forming strategic partnerships including Microsoft Azure, SAP Analytics, Snowflake, Matillion and many more, leveraging the strength of our cloud and network...

  • AWS Data Analyst

    2 weeks ago

    Sydney, New South Wales, Australia The Data Foundry Full time

    Job DescriptionYour impact As an AWS Data Analyst, you will play a key role in working with customers to understand their business requirements, map these to their data capabilities, liaise with our internal Data Engineers to design and build elegant solutions, then partner with our customers to deploy those solutions in a way that allows them to turn data...

  • Azure Devops Engineer

    1 week ago

    Sydney, New South Wales, Australia Data#3 Full time

    Engineering - Software (Information & Communication Technology) Full time Add expected salary to your profile for insights Join Microsoft's largest Australian Partner HRD Employer of Choice and Great Place to Work Brisbane, Sydney or Melbourne Based - Hybrid Working As an Azure Dev Ops Engineer you will work across multiple organisations, managing the latest...

  • Ms Engineer

    1 week ago

    Sydney, New South Wales, Australia Ntt Data Full time

    NTT DATA NTT DATA helps clients transform through consulting, industry solutions, business process services, IT modernization and managed services. View company page Want to be a part of our team?The Security Managed Services Engineer (L4) is responsible for ensuring a managed service is provided to all clients, to ensure that their Security infrastructure...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia HiTech Group Full time

    Leading Federal Government Department Initial 12 month contract + 12 month extension Rewarding daily rates $$$ A dynamic Federal Government Department is looking to engage an experienced Data Engineer to be apart of the department's internal key data centric projects. You will be responsible demonstrating strong data engineering and analytics skills with...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia HiTech Group Full time

    Exciting opportunity within a Federal Government Department Initial 12 month contract with EXTENSION options Rewarding hourly rates $$$ One of Australia's principal Federal Government departments is looking to engage a skilled Data Engineer & Data Modeller to assist in collaborating with key business areas in the development and implementation of consumable...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Experis ManpowerGroup Sp. z o.o. Full time

    Our government client are looking for several data engineers to join their team on a contract basis.Please note only Australian Citizens will be considered as security clearance is required.Role Responsibilities and DutiesDesign, construct, and optimize data lakes, CI/CD, and Synapse pipelines within the Azure stack.Develop scalable architecture practices...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Atlassian Full time

    We are looking for a Data Engineer to join our UIP - Frontend Observability team.The Frontend Observability team is part of the UI Platform organisation.Our team's mission is to help teams build fast experiences and make sure they stay fast. We build tools for measuring, monitoring and catching regressions in frontend performance.UI Platform delivers...

  • Data Engineering

    1 week ago

    Sydney, New South Wales, Australia Randstad Full time

    About the company: My client is a leading global multinational investment banking and financial services company. You will be joining the data engineering team to help build the best technology company in the financial services industry. You will play a leading role in shaping and driving the next evolution of their data and analytics platform, providing...

  • data engineer

    1 week ago

    Sydney, New South Wales, Australia Opus Recruitment Solutions Full time

    ApplyData Engineer Sydney, Australia Permanent$ $130000 per annum Are you a skilled Data Engineer looking for your next challenge? I am currently seeking a Senior Data Engineer with expertise in Azure, Azure Data Factory, Databricks, Synapse, Power BI, and strong experience in SQL to join my clients dynamic team.Responsibilities:Designing, building, and...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Practiv Limited Full time

    As a Practiv Data Engineer you will join a team of passionate data-cloud technologists who help deliver end to end data engineering solutions for our customers from building data pipelines, warehousing the data and visualisation/presentation. Practiv Data Engineers understand both data pipeline development, operations and management as well as delivery of...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Practiv Full time

    As a Practiv Data Engineer you will join a team of passionate data-cloud technologists who help deliver end to end data engineering solutions for our customers from building data pipelines, warehousing the data and visualisation/presentation. Practiv Data Engineers understand both data pipeline development, operations and management as well as delivery of...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Opus Recruitment Solutions Ltd Full time

    Apply Data EngineerSydney, AustraliaPermanent$ $130000 per annumAre you a skilled Data Engineer looking for your next challenge? I am currently seeking a Senior Data Engineer with expertise in Azure, Azure Data Factory, Databricks, Synapse, Power BI, and strong experience in SQL to join my clients dynamic team.Responsibilities:Designing, building, and...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Data Engineer Sydney $175,000 Inc Super + BonusAs a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment.Key...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Talenza Full time

    Data Engineer Sydney $170,000 Inc Super + Bonus As a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment. Key...

  • Data Engineers

    1 week ago

    Sydney, New South Wales, Australia CYOS Solutions Full time

    Application closing date: Friday, 14 June 2024 11:59pm, Canberra time Estimated start date: Monday, 01 July 2024 Location of work: NSWWorking arrangements: The team is based in Canberra however, working from home arrangements are considered subject to home-based work application and suitability.Interstate candidates will also be considered.However, there...

  • Data Engineers

    1 week ago

    Sydney, New South Wales, Australia CYOS Solutions Full time

    Application closing date: Friday, 14 June 2024 11:59pm, Canberra timeEstimated start date: Monday, 01 July 2024Location of work:NSWWorking arrangements: The team is based in Canberra however, working from home arrangements are considered subject to home-based work application and suitability.Interstate candidates will also be considered.However, there will...

  • Data Engineers

    4 weeks ago

    Sydney, New South Wales, Australia CYOS Solutions Full time

    Application closing date: Friday, 14 June 2024 • 11:59pm, Canberra timeEstimated start date: Monday, 01 July 2024Location of work: NSWWorking arrangements: The team is based in Canberra however, working from home arrangements are considered subject to home-based work application and suitability. Interstate candidates will also be considered. However, there...

  • Data Engineers

    1 week ago

    Sydney, New South Wales, Australia CYOS Solutions Full time

    Application closing date: Friday, 14 June 2024 • 11:59pm, Canberra timeEstimated start date: Monday, 01 July 2024Location of work: NSWWorking arrangements: The team is based in Canberra however, working from home arrangements are considered subject to home-based work application and suitability. Interstate candidates will also be considered. However, there...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Kindred Group plc Full time

    The roleWe are looking for a motivated Data Engineer to join our team and support the Data Engineering pipeline design and implementation experience within a commercial environment.As a listed and regulated company, Kindred Group pays great attention to player safety and sustainability. Our purpose is to transform gambling by being a trusted source of...