Data Engineer with Python

4 weeks ago


Canberra, Australia Unison Consulting Pte Ltd Full time

  • Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
  • Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
  • Optimize and troubleshoot AWS Glue jobs for performance and reliability.
  • Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.
  • Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
  • Apply data cleansing, enrichment, and validation techniques to ensure data quality and accuracy.
  • Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
  • Optimize ETL workflows for performance and efficiency.
  • Collaborate with data architects to design and implement data models that support business requirements.
  • Ensure data structures are optimized for analytics and reporting.
  • Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.
  • Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
  • Implement indexing, partitioning, and other database optimization techniques.
  • Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
  • Work closely with software engineers to integrate data solutions into larger applications.
  • Implement monitoring solutions to track data pipeline performance and proactively identify and address issues
  • Ensure compliance with data privacy regulations and company policies.
  • Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.

Requirements

  • Proficiency in Python and PySpark.
  • Strong knowledge of data engineering concepts and best practices.
  • Hands-on experience with AWS Glue and other AWS services.
  • Experience with big data technologies and distributed computing.
  • Familiarity with database management systems (SQL and NoSQL).
  • Understanding of ETL processes and data modeling.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.
  • Bachelor's degree in Computer Science, Information Technology, or a related field.

  • Python Developer

    4 weeks ago


    Canberra, ACT, Australia Randstad Australia Full time

    Multi-year engagement Canberra basedCandidate must be an Australian citizen with a Baseline security clearance Randstad Digital (Formerly Finite Recruitment) Randstad Digital was announced under the umbrella of Randstad, the world's largest talent company and a partner of choice to clients. Our 46,000 employees around the world make a positive impact on...

  • Data Analyst

    4 weeks ago


    Canberra, Australia HiTech Personnel Full time

    **Reference #**: - JF/JJ2302**Title**: - Data Analyst - SQL, Power BI, Python**Category**: - ICT**Location**: - ACT**Work Type**: - Contract**Remuneration**: - $Neg**Term**: - 12 months + 24 month**Description**: - **High Profile Federal Government Department!**: - **Initial contract 12 month with 2 x 12 month extension options!**: - **Rewarding...

  • Software Engineer

    3 weeks ago


    Canberra, Australia Yeah! Global Full time

    Job Description This is a remote position.One of the clients is looking to extend its technical team by 7 Software engineers globally. This is a 100% remote role.Note: Please share your resume in English only.We are looking for a Software Engineer who is passionate about solving large business optimization problems at scale. If you are comfortable working...

  • Data Modeller

    4 weeks ago


    Canberra, Australia Talent International Full time

    australia australian capital territory contract negotiable- **Seeking experienced Data Modellers**: - **12 months + 12 months**: - **Canberra location**: - **Must be able to provide proof of Aus Citizenship** **Our Client** Our client is highly respected organisation, who provides services and supports to thousands of individuals around Australia. With...

  • Software Engineer

    3 weeks ago


    Canberra, Australia Yeah! Global Full time

    This is a remote position.One of the clients is looking to extend its technical team by 7 Software engineers globally. This is a 100% remote role.Note: Please share your resume in English only.We are looking for a Software Engineer who is passionate about solving large business optimization problems at scale. If you are comfortable working in a start-up...

  • Data Analyst

    4 weeks ago


    Canberra, Australia Emanate Technology Full time

    Emanate Technology is working with a key client to seek the services of a skilled Data Analyst. As part of this role the Data Analyst will be reporting to the Director and will be responsible for gathering, cleaning, manipulating and integrating data from a range of sources to produce new datasets and visualisations.The preferred candidate will be working in...

  • El1 Data Analyst

    4 weeks ago


    Canberra, Australia face2face Recruitment Full time

    **Closing Date**:9 am Monday 29 January 2024 **face2face reference: 596353** **Location: Canberra, ACT or Remote** **Hours: full-time** **Entry requirements: Australian Citizen + Ability to obtain Negative Vetting 1** **Remuneration: $105,644.00 plus super (prorated)** **About the role**: We are looking for a Data Analyst with at least 6 years of...

  • Data Scientist

    4 weeks ago


    Canberra, Australia ignite Full time

    Location : Sydney, Canberra. Duration : 12-month contract with 1x12 month extension. Eligibility : Australian citizenship with NV1 Security Clearance required. In office and working from home arrangements can be discussed. The Role: Seeking a Data Scientist for a contract opportunity, where you will work with one or more multi-disciplinary teams to...


  • Canberra, Australia Tencent Full time

    Responsibilities: Responsibilities●Responsible for the design and coding of support services in the process of game development and operation●Responsible for global deployment and maintenance of services●Collaborate with data and AI engineers to build large-scale data processing systems Requirements: Qualifications●Bachelor's degree or...

  • Data Analyst

    4 weeks ago


    Canberra, Australia Talent International Full time

    **Job Details**: **Location** Canberra **Salary** Negotiable **Job Type** Contract **Ref** BBBH100873_1681713848 **Contact** Yvonne Yang **Posted** 32 minutes ago - ** Canberra located and multiple positions are available** **About the role** You will be responsible for extracting and integrating data from a variety of sources to develop and...

  • Data Analyst

    4 days ago


    Canberra, Australia Agylex-Sprinx Global Full time

    One of the **Esteemed Federal Clients **in **Canberra, ACT **is hiring a **"Data Analyst - Visual Analytics" **for a duration contract. **Title - **Data Analyst - Visual Analytics **Location-** ACT **Working Arrangement - **Onsite **Opportunity type - **Contract **Australian Citizens with Baseline are applicable for this role.** **Requirements**: The...

  • Data Analyst

    7 days ago


    Canberra, Australia Australian National University Full time

    **Classification**: ANU Officer 6/7 (Administration) **Salary package**: $86,960 - $100,098 per annum plus 17% superannuation **Terms**: Full time, Continuing (Contingent Funded) (until the 31 Dec 2023, with possibility of extension) **The Position** The Data Analyst plays a key role in demonstrating CEAT’s contribution to supporting and sustaining an...


  • Canberra, Australia Talent International Full time

    australia australian capital territory contract negotiable- Exciting opportunity for multiple SQL Data Analyst - 12 Month contract + extensions - Canberra located **The Client** Our client is highly respected organisation, who provides services and supports to thousands of individuals around Australia. With multiple locations around Canberra, this vital...


  • Canberra, Australia Allura Partners Full time

    ClouderaDataAzure Cloudera Data Platform Engineer - Contract, $1,200 to $1,300 a day / NV1 - NV2I am working with a Government department who are looking for an engineer with Cloudera experience for a new projectIn the role you will be responsible for the Cloudera Data platform and have experience in the following:- Experience in deployment of Cloudera Data...

  • Data Analyst

    4 days ago


    Canberra, Australia Department of Industry, Science, Energy & Resources Full time

    **Job no**: 512593 **Employment type**: Ongoing **Location**: Canberra, Sydney, Brisbane, Adelaide, Melbourne, Perth, Any location in Australia **Categories**: APS Level 6, Data **About the department**: The department plays a key role in the Australian Government’s agenda to create jobs and build a stronger, more resilient and competitive economy. Our...

  • Storage Specialist

    6 days ago


    Canberra, Australia Talent International Full time

    australia australian capital territory permanent au$95000 - au$105000 per annum + + 15.4% superannuation- **Canberra based only**: - **Permanent position within a scientific organisation**: - **$95-$105k + 15.4% Superannuation**: - **Citizenship minimum requirement - NV1 clearance will be obtained** As a **Storage Engineer** working for Australia’s...


  • Canberra, Australia Australian Government Full time

    Australian Public Service - Brindabella Park, ACT - 8- 12 April 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...


  • Canberra, Australia Australian Government Full time

    Australian Public Service - Brindabella Park, ACT - 8-12 July 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...


  • Canberra, Australia Australian Government Full time

    Australian Public Service - Brindabella Park, ACT - 8 - 12 April 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...


  • Canberra, Australia Australian Government Clean Energy Regulator Full time

    Canberra Based - Ongoing/Non-Ongoing (Non-ongoing will be offered for 12 months) - Scheme Support Division **About the Section**: The Data and Innovation section supports the agency to unlock the potential of our data to drive innovation and compliance for the agency. Specific responsibilities include: - Support the Chief Data Officer and the agency in...