Data Engineer with using Apache Spark, Python, Scala

10 hours ago

Melbourne, Victoria, Australia Tech Aalto Full time $120,000 - $180,000 per year
Data Engineer with using Apache Spark, Python, Scala.

Location: Melbourne

Position Type: Contract

Data Engineer

 As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.

 Key Responsibilities:
  • Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
  • Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
  • Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
  • Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.
  • Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
  • Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
  • Implement and adhere to best practices for data governance, security, and compliance.
  • Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for dataworkflows.
  • Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.
 Requirements:
  • Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
  • Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
  • Experience with performance tuning and optimization of Spark jobs and SQL queries.
  • Familiarity with stream processing frameworks like Kafka and messaging systems.
  • Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
  • Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration skills, with the ability to work effectively in a team environment
​​​​​​​When you apply, you voluntarily consent to the disclosure, collection and use of your personal data for employment/recruitment and related purposes in accordance with the Tech Aalto Privacy Policy, a copy of which is published at Tech Aalto's website )

Confidentiality is assured, and only shortlisted candidates will be notified for interviews.

 

  • Senior Data Engineers

    2 weeks ago

    Melbourne, Victoria, Australia Otic Group Full time $120,000 - $180,000 per year

    "Otic" means smart people doing smart work, together.We are a wholly owned Australian company committed to helping our clients design and build intelligent software solutions that unlock value in their business.Otic Group was formed to provide talented technology professionals an opportunity to work with not only some of the most prominent companies in...

  • Senior Data Engineer

    13 hours ago

    Melbourne, Victoria, Australia Ippon Technologies Full time $120,000 - $180,000 per year

    Ippon Australia is a team of true technologists who leverage contemporary thinking and tools to enhance the way our clients drive value. Whether it's a bespoke platform to improve employee engagement within a national retail chain, or developing an app that provides next level direct customer interaction; value is delivered to our clients when we begin by...

  • AWS Consultant

    12 hours ago

    Melbourne, Victoria, Australia Zone IT Solutions Full time $120,000 - $180,000 per year

    We are seeking an experienced AWS Databricks Architect. In this role, you will lead the design and implementation of data solutions on the AWS platform using Databricks, ensuring optimal performance and scalability.RequirementsMinimum of 5 years of experience in data architecture and data engineering. Strong expertise in AWS services, particularly with AWS...

  • Data Bricks

    10 hours ago

    Melbourne, Victoria, Australia Pyramid Global Technologies Full time $120,000 - $180,000 per year

    "Tech Stack Required:Required: 12+ Years· Experience in Data Lake, Azure Data Bricks· Experience Designing and deploying Databricks platforms for any of AWS, Azure or Google Cloud· Experience building and deploying data engineering pipelines into production, including using automation best practices for CI/CD· Guiding clients as they implement...

  • Senior Data Engineer

    2 weeks ago

    Melbourne, Victoria, Australia Maincode Full time $120,000 - $180,000 per year

    OverviewMaincode is building sovereign AI models in Australia. We are training foundation models from scratch, designing new reasoning architectures, and deploying them on state-of-the-art GPU clusters. Our models are built on datasets we create ourselves, curated, cleaned, and engineered for performance at scale. This is not buying off-the-shelf corpora or...

  • Senior Data Engineer

    14 hours ago

    Melbourne, Victoria, Australia Maincode Full time $120,000 - $180,000 per year

    OverviewMaincode is building sovereign AI models in Australia. We are training foundation models from scratch, designing new reasoning architectures, and deploying them on state-of-the-art GPU clusters. Our models are built on datasets we create ourselves, curated, cleaned, and engineered for performance at scale. This is not buying off-the-shelf corpora or...

  • AWS Data Engineer

    1 week ago

    Melbourne, Victoria, Australia Resource Corner Full time $120,000 - $180,000 per year

    Role - Data EngineerLocation - SydneyContract positionHandson Experience ( at least 3-4 years with the skills mentioned with overall 8 to 12 years as Data Engineer) -Apache Flink, Amazon Managed Streaming for Kafka ( MSK) Apache Kafka,Preferably Banking Domain and AWS backgroundSkills:8 to 12 years as Data EngineerApache FlinkApache KafkaAWS

  • Python & Data Engineer

    2 days ago

    Melbourne, Victoria, Australia BlueRose Technologies Full time $90,000 - $120,000 per year

    Job Details:We are seeking a skilled professional with strong technical expertise in Python, SQL, and data automation. The ideal candidate should be detail-oriented, proactive, and capable of working closely with business stakeholders to deliver data-driven automation and analytical solutions.Must Have Skills:Python programming with hands-on experience in...

  • Data Engineer

    2 days ago

    Melbourne, Victoria, Australia Nuage Technology Group Full time $80,000 - $120,000 per year

    Nuage has partnered with one of Melbourne's highest-performing data teams to hire a DataOps Engineer to support and evolve a large-scale Azure data platform. You'll be working across enterprise data pipelines, IoT feeds, Databricks workloads and alerting/incident response, blending platform engineering, operational reliability, and hands-on pipeline...

  • Senior Data Engineer

    1 week ago

    Melbourne, Victoria, Australia Turing Consulting Full time $120,000 - $150,000 per year

    Role Title: Senior Data EngineerLocation: Melbourne, Victoria, AustraliaContract Duration: 4 months (extendable)Description:Technical:Pyspark, Python, SparkSQL, SQL and Glue.AWS cloud experienceGood understanding of dimensional modellingGood understanding DevOps, CloudOps, DataOps, CI/CD & with a SRE mindsetUnderstanding of Lakehouse and DW...