Apache Spark Data Engineer

2 weeks ago

Sydney, New South Wales, Australia Itbility Full time
  • Apache Spark Data Engineer
  • Sydney
  • Permanent

Job Description :

Should have created Azure solutions and enabled Azure capabilities

· Should have hands-on experience on Function Apps & Web Apps

· Should have hands-on experience on Synapse(Spark), ADLS2, Purview

· Should know one of the programming languages which is either Python or Scala

· Good to have linux scripting and powershell experience

· Should have source version control experience in bitbucket, git, Jenkins and DevSecOps

· Ability to identify existing gaps at customer location and provide apt extensible solutions across various Azure services, mainly Azure Purview, Function App and Web Services.

· Ability to interact with wider teams inclusive of Platform, Engineering, Framework and wider technical teams.

· Ability to convert given business requirements to technical specification in line to customer current architecture and initiatives.

· Designing the documents such as Software Technical Solution(STS), Detail Design Approach(DDA), Impact Analysis of the project, Detailed Technical Design( DTD), Mapping Specifications based on the Business requirements document(BRD), Interface Specifications.

· Estimating the project efforts and project planning

· Perform the Impact Analysis to the impacted and the downstream applications.

· Unit Testing of developed code, diagnosing, correcting errors and faults as per agreed guidelines and quality standards.

· Software development support for production implementation and warranty defect fixes.

· Reviewing the code to ensure the Coding and Quality standards established are achieved

· Participating in Daily scrum stand-up's, Sprint Planning, PI Planning, Demos and retrospective Agile ceremonies.

· Technically support the team to deliver high quality deliverable to meet/exceed SLAs

· Resource on-boarding, training, Team Mentoring and Conflict resolution.

All candidates should have full working rights in Australia.

Only shortlisted candidates will be contacted for this role.

To apply, please submit your resume ASAP for immediate consideration or email


#J-18808-Ljbffr

  • Data Engineer- Hadoop Big Data Platform

    2 weeks ago

    Sydney, New South Wales, Australia Virtusa Full time

    Data Engineer- Hadoop Big Data Platform Description Implementation experience in BigData Platform ; preferably in Cloudera Hadoop platform Minimum 2 years of Development experience using Hadoop eco system tools & utilities: MapReduce, Spark, Kafka, Sqoop, Impala, Hive etc Ability to work independently and also contribute to overall architecture and...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Practiv Limited Full time

    As a Practiv Data Engineer you will join a team of passionate data-cloud technologists who help deliver end to end data engineering solutions for our customers from building data pipelines, warehousing the data and visualisation/presentation. Practiv Data Engineers understand both data pipeline development, operations and management as well as delivery of...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Practiv Full time

    As a Practiv Data Engineer you will join a team of passionate data-cloud technologists who help deliver end to end data engineering solutions for our customers from building data pipelines, warehousing the data and visualisation/presentation. Practiv Data Engineers understand both data pipeline development, operations and management as well as delivery of...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Talenza Full time

    Data Engineer Sydney $170,000 Inc Super + Bonus As a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment. Key...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Data Engineer Sydney $175,000 Inc Super + BonusAs a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment.Key...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Lendaly Pty Full time

    Improve upon existing systems that collect, manage and convert data for our teams to interpret in a meaningful way. Experience with building AI/ML models and rule-based engines required.ResponsibilitiesDesign, build, and maintain scalable data pipelines to collect, process, and store large volumes of structured and unstructured data from various...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Atlassian Full time

    We are looking for a Data Engineer to join our UIP - Frontend Observability team.The Frontend Observability team is part of the UI Platform organisation.Our team's mission is to help teams build fast experiences and make sure they stay fast. We build tools for measuring, monitoring and catching regressions in frontend performance.UI Platform delivers...

  • Principal, Data

    2 weeks ago

    Sydney, New South Wales, Australia Practiv Limited Full time

    As a Principal of our Data & Analytics practice, you will join a team of passionate data-cloud technologists who help deliver end to end data engineering solutions for our customers from building data pipelines, warehousing the data and visualisation/presentation.Practiv Data Engineers understand both data pipeline development, operations and management as...

  • Mid Level Data engineer

    2 weeks ago

    Sydney, New South Wales, Australia Clarrow Full time

    IT & TelecommsSoftware - DeveloperSydneyPermanent / Full TimeStart Date: 20/5/2024World Leading Financial InstitutionSydney CBD LocationExcellent Salary and Bonus Scheme OfferedWe are looking for a Mid Level Data Engineer to join the global client in developing innovative applications and analysis for a migration project within their commodities global...

  • Data Engineering Manager

    2 weeks ago

    Sydney, New South Wales, Australia The Argyle Network Full time

    The Argyle NetworkSydney, AustraliaPosted 9 hours ago Permanent Competitive Salary + Bonus Are you a Data Engineering Manager or Tech Lead with experience in endtoend transformations? Join our client an Aussie Technology BankThe Client:Leading Aussie Bank dedicated to innovation and collaboration. Having invested in the latest cutting edge technologies,...

  • Python Engineer

    2 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Python Engineer Sydney $175,000 Inc Super + Bonus As a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives.This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment.Key...

  • Python Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Talenza Full time

    Python Engineer Sydney $170,000 Inc Super + Bonus As a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment. Key...

  • Python Engineer

    2 weeks ago

    Sydney, New South Wales, Australia ClearCompany Full time

    Python Engineer Sydney $175,000 Inc Super + BonusAs a Data Engineer within our dynamic Engineering team, you will play a crucial role in developing cutting-edge applications to support our Data Transformation initiatives. This is a permanent position offering the opportunity to work on innovative projects within a supportive team environment.Key...

  • Data Engineer, Sydney

    2 weeks ago

    Sydney, New South Wales, Australia GrainCorp Full time

    Press Tab to Move to Skip to Content Link Select how often (in days) to receive an alert: Create Alert Select how often (in days) to receive an alert: About GrainCorpWhat did you have for breakfast today? Whether it's the flour in your toast or the grain in your cereal, it's highly likely that GrainCorp helped get it onto your plate As we find new ways to...

  • Senior Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Kangan Institute Full time

    Location: Flexible across metro or regional locationsRole Type: Full timeBendigo Kangan Institute is one of Victoria's largest TAFE providers. We offer state-of-the-art training centres providing a wide range of courses in a variety of fields. We foster a culture that is customer-centric, works together and focuses on achieving our best in a healthy and...

  • Principal Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Culture Amp Full time

    How you can help make a better world of work Culture Amp is on a mission to embed analytics and insights across our product suite.We have recently acquired the people analytics platform Orgnostic, and are looking to build out our data platform to further unlock the value of our data.To help us deliver on this mission, we are seeking a Principal Data Engineer...

  • Big Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Pyramid Global Technologies Full time

    About the job Big Data Engineer Job Description:· 5+ years experience working in Data Engineering and Warehousing. · 3 -5 years' experience integrating data into analytical platforms · Experience in ingestion technologies (e.g. Sqoop, flume), processing technologies (Spark/Scala), and storage (e.g. HDFS, HBase, Hive) · Experience in data profiling,...

  • Lead Data Modeler

    2 weeks ago

    Sydney, New South Wales, Australia Cognizant Technology Solutions Full time

    Permanent full-time opportunity in Sydney Join a dynamic and high performing team driven by positive values Leverage your solutioning experience and "can do" attitudeAs a Lead Data Modeler, you will be responsible to lead & design Big data E2 E implementation. You will collaborate with Enterprise & business architects to define big data implementations and...

  • Senior Data Engineer – 2 roles available. Daily rate contracts

    2 weeks ago

    Sydney, New South Wales, Australia The Recruitment Company Full time

    Australian Fintech seeks two Data Engineers (6 month Contract initially) to work on a new project to work on a key integration project. You will work on a friendly team with cutting-edge technology. Great chance to work with Databricks and use some of the newest Databricks features Role:Work collaboratively as part of the Data Squad to develop new...

  • Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Bancroft School Full time

    The Data Engineer is responsible for building and supporting the big data platform.As a Data Engineer, your responsibilities include: Develop, test and support future-ready data solutions for customers across industry verticals Develop, test and support end-to-end batch and near real-time data flows/pipelines Demonstrate understanding in data architectures,...