Data Engineer with Python
4 weeks ago
- Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
- Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
- Optimize and troubleshoot AWS Glue jobs for performance and reliability.
- Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.
- Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
- Apply data cleansing, enrichment, and validation techniques to ensure data quality and accuracy.
- Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
- Optimize ETL workflows for performance and efficiency.
- Collaborate with data architects to design and implement data models that support business requirements.
- Ensure data structures are optimized for analytics and reporting.
- Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.
- Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
- Implement indexing, partitioning, and other database optimization techniques.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
- Work closely with software engineers to integrate data solutions into larger applications.
- Implement monitoring solutions to track data pipeline performance and proactively identify and address issues
- Ensure compliance with data privacy regulations and company policies.
- Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.
Requirements
- Proficiency in Python and PySpark.
- Strong knowledge of data engineering concepts and best practices.
- Hands-on experience with AWS Glue and other AWS services.
- Experience with big data technologies and distributed computing.
- Familiarity with database management systems (SQL and NoSQL).
- Understanding of ETL processes and data modeling.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills.
- Bachelor's degree in Computer Science, Information Technology, or a related field.
-
Python Developer
4 weeks ago
Canberra, ACT, Australia Randstad Australia Full timeMulti-year engagement Canberra basedCandidate must be an Australian citizen with a Baseline security clearance Randstad Digital (Formerly Finite Recruitment) Randstad Digital was announced under the umbrella of Randstad, the world's largest talent company and a partner of choice to clients. Our 46,000 employees around the world make a positive impact on...
-
Data Analyst
4 weeks ago
Canberra, Australia HiTech Personnel Full time**Reference #**: - JF/JJ2302**Title**: - Data Analyst - SQL, Power BI, Python**Category**: - ICT**Location**: - ACT**Work Type**: - Contract**Remuneration**: - $Neg**Term**: - 12 months + 24 month**Description**: - **High Profile Federal Government Department!**: - **Initial contract 12 month with 2 x 12 month extension options!**: - **Rewarding...
-
Software Engineer
3 weeks ago
Canberra, Australia Yeah! Global Full timeJob Description This is a remote position.One of the clients is looking to extend its technical team by 7 Software engineers globally. This is a 100% remote role.Note: Please share your resume in English only.We are looking for a Software Engineer who is passionate about solving large business optimization problems at scale. If you are comfortable working...
-
Data Modeller
4 weeks ago
Canberra, Australia Talent International Full timeaustralia australian capital territory contract negotiable- **Seeking experienced Data Modellers**: - **12 months + 12 months**: - **Canberra location**: - **Must be able to provide proof of Aus Citizenship** **Our Client** Our client is highly respected organisation, who provides services and supports to thousands of individuals around Australia. With...
-
Software Engineer
3 weeks ago
Canberra, Australia Yeah! Global Full timeThis is a remote position.One of the clients is looking to extend its technical team by 7 Software engineers globally. This is a 100% remote role.Note: Please share your resume in English only.We are looking for a Software Engineer who is passionate about solving large business optimization problems at scale. If you are comfortable working in a start-up...
-
Data Analyst
4 weeks ago
Canberra, Australia Emanate Technology Full timeEmanate Technology is working with a key client to seek the services of a skilled Data Analyst. As part of this role the Data Analyst will be reporting to the Director and will be responsible for gathering, cleaning, manipulating and integrating data from a range of sources to produce new datasets and visualisations.The preferred candidate will be working in...
-
El1 Data Analyst
4 weeks ago
Canberra, Australia face2face Recruitment Full time**Closing Date**:9 am Monday 29 January 2024 **face2face reference: 596353** **Location: Canberra, ACT or Remote** **Hours: full-time** **Entry requirements: Australian Citizen + Ability to obtain Negative Vetting 1** **Remuneration: $105,644.00 plus super (prorated)** **About the role**: We are looking for a Data Analyst with at least 6 years of...
-
Data Scientist
4 weeks ago
Canberra, Australia ignite Full timeLocation : Sydney, Canberra. Duration : 12-month contract with 1x12 month extension. Eligibility : Australian citizenship with NV1 Security Clearance required. In office and working from home arrangements can be discussed. The Role: Seeking a Data Scientist for a contract opportunity, where you will work with one or more multi-disciplinary teams to...
-
Backend Development intern
4 weeks ago
Canberra, Australia Tencent Full timeResponsibilities: Responsibilities●Responsible for the design and coding of support services in the process of game development and operation●Responsible for global deployment and maintenance of services●Collaborate with data and AI engineers to build large-scale data processing systems Requirements: Qualifications●Bachelor's degree or...
-
Data Analyst
4 weeks ago
Canberra, Australia Talent International Full time**Job Details**: **Location** Canberra **Salary** Negotiable **Job Type** Contract **Ref** BBBH100873_1681713848 **Contact** Yvonne Yang **Posted** 32 minutes ago - ** Canberra located and multiple positions are available** **About the role** You will be responsible for extracting and integrating data from a variety of sources to develop and...
-
Data Analyst
4 days ago
Canberra, Australia Agylex-Sprinx Global Full timeOne of the **Esteemed Federal Clients **in **Canberra, ACT **is hiring a **"Data Analyst - Visual Analytics" **for a duration contract. **Title - **Data Analyst - Visual Analytics **Location-** ACT **Working Arrangement - **Onsite **Opportunity type - **Contract **Australian Citizens with Baseline are applicable for this role.** **Requirements**: The...
-
Data Analyst
7 days ago
Canberra, Australia Australian National University Full time**Classification**: ANU Officer 6/7 (Administration) **Salary package**: $86,960 - $100,098 per annum plus 17% superannuation **Terms**: Full time, Continuing (Contingent Funded) (until the 31 Dec 2023, with possibility of extension) **The Position** The Data Analyst plays a key role in demonstrating CEAT’s contribution to supporting and sustaining an...
-
Assistant Director of Data and Reporting
7 days ago
Canberra, Australia Talent International Full timeaustralia australian capital territory contract negotiable- Exciting opportunity for multiple SQL Data Analyst - 12 Month contract + extensions - Canberra located **The Client** Our client is highly respected organisation, who provides services and supports to thousands of individuals around Australia. With multiple locations around Canberra, this vital...
-
Cloudera Data Platform Engineer
2 days ago
Canberra, Australia Allura Partners Full timeClouderaDataAzure Cloudera Data Platform Engineer - Contract, $1,200 to $1,300 a day / NV1 - NV2I am working with a Government department who are looking for an engineer with Cloudera experience for a new projectIn the role you will be responsible for the Cloudera Data platform and have experience in the following:- Experience in deployment of Cloudera Data...
-
Data Analyst
4 days ago
Canberra, Australia Department of Industry, Science, Energy & Resources Full time**Job no**: 512593 **Employment type**: Ongoing **Location**: Canberra, Sydney, Brisbane, Adelaide, Melbourne, Perth, Any location in Australia **Categories**: APS Level 6, Data **About the department**: The department plays a key role in the Australian Government’s agenda to create jobs and build a stronger, more resilient and competitive economy. Our...
-
Storage Specialist
6 days ago
Canberra, Australia Talent International Full timeaustralia australian capital territory permanent au$95000 - au$105000 per annum + + 15.4% superannuation- **Canberra based only**: - **Permanent position within a scientific organisation**: - **$95-$105k + 15.4% Superannuation**: - **Citizenship minimum requirement - NV1 clearance will be obtained** As a **Storage Engineer** working for Australia’s...
-
Asd Cyber Security and Robotics Work Experience
4 weeks ago
Canberra, Australia Australian Government Full timeAustralian Public Service - Brindabella Park, ACT - 8- 12 April 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...
-
Asd Cyber Security and Robotics Work Experience
4 weeks ago
Canberra, Australia Australian Government Full timeAustralian Public Service - Brindabella Park, ACT - 8-12 July 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...
-
Asd Cyber Security and Robotics Work Experience
4 weeks ago
Canberra, Australia Australian Government Full timeAustralian Public Service - Brindabella Park, ACT - 8 - 12 April 2024 The Australian Signals Directorate (ASD) is the national authority for information security and through its Work Experience Program offers students from **Years 11-12** an opportunity to extend their knowledge and skills in science, technology, engineering and mathematics (STEM), with a...
-
Assistant Manager- Data Science
6 days ago
Canberra, Australia Australian Government Clean Energy Regulator Full timeCanberra Based - Ongoing/Non-Ongoing (Non-ongoing will be offered for 12 months) - Scheme Support Division **About the Section**: The Data and Innovation section supports the agency to unlock the potential of our data to drive innovation and compliance for the agency. Specific responsibilities include: - Support the Chief Data Officer and the agency in...