Current jobs related to Pyspark Data engineer - Melbourne - Capgemini
-
Tech Lead
3 months ago
Melbourne, Australia Snaphunt Full timeThe OfferGreat Work CultureThe JobYou will be responsible for : Building new features on our data platform primarily in PySpark.Drive best practice for data engineering and site reliability.Challenge the development team on their delivery of quality, unit test coverage, and assistance in the creation of development artefacts.The ProfileRequirements10+ years...
-
Tech Lead
3 months ago
Melbourne, Australia SkyBeam Technology Full timeSkyBeam Technology is an Australia Based professional services company that design and deliver tailored workforce solutions to enable businesses to experience the benefits of managing a workforce in the evolving world. Our workforce solutions specialised in: Recruitment / Sourcing Services, Contract management services, Outsourced Payroll Services and HR...
-
Data Engineer
4 weeks ago
Melbourne, Victoria, Australia Profusion Group Full timeJob Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Profusion Group. As a Data Engineer, you will play a critical role in designing and implementing advanced data solutions on AWS and related technologies.Key Responsibilities:Design and Implement Data Solutions: Collaborate with cross-functional teams to design and...
-
Data Engineer
1 month ago
Melbourne, Australia Profusion Group Full timeDesign scalable data solutions using AWS, PySpark, and Databricks Drive innovation in data architecture with a focus on advanced cloud and big data technologies Collaborate with cross-functional teams to deliver high-performance data platforms Company Overview A leader in wealth management sector is driving innovation through data engineering...
-
Data Engineering Lead
2 months ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology, a leading professional services company in Australia, is seeking a highly skilled professional to lead our data engineering efforts.About the RoleWe are looking for a seasoned expert to build and maintain our data platform using PySpark, drive best practices for data engineering and site reliability, and collaborate with the development...
-
Data Engineering Lead
1 month ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology, a leading professional services company in Australia, is seeking a highly skilled Technical Lead - Data to join their team.About the RoleThe successful candidate will be responsible for building and maintaining our data platform using PySpark, driving best practices for data engineering and site reliability, and collaborating with the...
-
Data Engineer
2 weeks ago
Melbourne, Victoria, Australia Profusion Group Full timeJob DescriptionJob Title: Data EngineerCompany: Profusion GroupJob Type: ContractLocation: RemoteJob Description:We are seeking a highly skilled Data Engineer to join our team at Profusion Group. As a Data Engineer, you will be responsible for designing and implementing scalable data solutions using AWS, PySpark, and Databricks.Key Responsibilities:Design...
-
Senior Data Engineering Lead
2 months ago
Melbourne, Victoria, Australia Snaphunt Full timeAbout the RoleWe are seeking a highly experienced Senior Data Engineering Lead to join our team at Snaphunt. As a key member of our data engineering team, you will be responsible for building and maintaining our data platform, primarily using PySpark.Key ResponsibilitiesDesign and implement scalable data pipelines using PySpark and Python.Drive best...
-
Data Engineering Lead
2 months ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology, a leading professional services company in Australia, is seeking a highly skilled professional to lead our data engineering efforts.About the RoleWe are looking for a seasoned expert to build and maintain our data platform using PySpark, drive best practices for data engineering and site reliability, and collaborate with the development...
-
Data Systems Engineer
2 months ago
Melbourne, Victoria, Australia Octopus Energy Full timeAbout the RoleWe're seeking a talented Data Engineer to join our team at Octopus Energy, a leading technology company in the energy sector. As a Data Engineer, you will play a crucial role in supporting the development of our data systems, working on core PySpark applications, dbt models, and data pipelines in Airflow.Key Responsibilities:Support the...
-
Data Engineering Lead
1 month ago
Melbourne, Victoria, Australia Snaphunt Full timeThe OpportunityWe are seeking a highly skilled Data Engineering Lead to join our team at Snaphunt. As a key member of our data platform, you will be responsible for building and maintaining our data infrastructure, driving best practices for data engineering and site reliability, and collaborating with the development team to ensure high-quality delivery.The...
-
Data Engineering Manager
3 weeks ago
Melbourne, Victoria, Australia Snaphunt Full timeThe OpportunityWe are seeking a highly skilled Data Engineering Manager to lead our data platform team in building and maintaining scalable data pipelines using PySpark.The RoleDesign and implement data pipelines using Python, PySpark, and SQL to support business growth.Collaborate with cross-functional teams to drive best practices for data engineering and...
-
Data Engineering Manager
2 weeks ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology is a professional services company that specializes in designing and delivering tailored workforce solutions to help businesses thrive in the evolving world.The RoleAs a key member of our team, you will be responsible for building new features on our data platform using PySpark, driving best practices for data engineering and site...
-
Data Engineering Leader
3 days ago
Melbourne, Victoria, Australia Snaphunt Full timeThe OpportunityAvoiding a monolithic architecture through incremental decomposition.The RoleAs a Data Engineering Leader, you will be responsible for:Designing and implementing scalable data pipelines using PySpark.Ensuring best practices for data engineering and site reliability.Collaborating with the development team to deliver high-quality solutions.The...
-
Data Engineering Manager
2 weeks ago
Melbourne, Victoria, Australia Snaphunt Full timeThe OpportunityWe are seeking a highly skilled Tech Lead to join our team at Snaphunt. As a key member of our data engineering team, you will be responsible for building and maintaining our data platform, driving best practices for data engineering and site reliability, and collaborating with the development team to deliver high-quality solutions.The...
-
Data Engineering Lead
4 weeks ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology, a leading professional services company in Australia, is seeking a highly skilled Technical Lead - Data to join their team.About the RoleThe successful candidate will be responsible for building new features on our data platform using PySpark, driving best practices for data engineering and site reliability, and challenging the...
-
Data Engineering Lead
4 weeks ago
Melbourne, Victoria, Australia SkyBeam Technology Full timeSkyBeam Technology, a leading professional services company in Australia, is seeking a highly skilled Technical Lead - Data to join their team.About the RoleAs a Technical Lead - Data, you will be responsible for building new features on our data platform primarily in PySpark, driving best practices for data engineering and site reliability, and challenging...
-
Data Engineer
1 month ago
Melbourne, Victoria, Australia Octopus Energy Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at Octopus Energy. As a Data Engineer, you will play a critical role in supporting the development of our core data systems, including our PySpark applications, dbt models, and data pipelines in Airflow. You will work closely with our data engineering team to design, build, and...
-
Data Architect
3 weeks ago
Melbourne, Victoria, Australia Profusion Group Full timeJob Title: Data EngineerCompany OverviewProfusion Group is a leader in the wealth management sector, driving innovation through data engineering and cloud infrastructure. As a Data Engineer, you will work with AWS, PySpark, and Databricks to develop data solutions that drive key business decisions.Role OverviewWe are seeking a highly skilled Data...
-
Data Architect
4 weeks ago
Melbourne, Victoria, Australia Profusion Group Full timeJob Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Profusion Group. As a Data Engineer, you will play a critical part in designing and implementing advanced data solutions on AWS and related technologies.Key Responsibilities:Logical Data Modeling: Perform logical data modeling and schema design, leveraging PySpark and...
Pyspark Data engineer
3 months ago
About Capgemini:
Capgemini is a diverse collective of more than 350,000 strategic and technological experts based across more than 50 countries, partnering with world-renowned clients to transform and manage their businesses. We are dedicated to leveraging cloud, data, AI, connectivity, software, digital engineering, and platforms to address the entire breadth of their business needs. This passion drives a powerful commitment - to unlock the true value of technology.
Over the last 18 months, we have tripled our business in Australia and New Zealand, with over 3,500 team members devoted to helping clients get the future they want. Now is the time to join our rapidly growing team who are at the forefront of finding new ways technology can help us reimagine what's possible, collecting unique career experiences with global brands and game-changing tech projects.
Let's talk about the team:
Our Insights and Data team helps our clients make better business decisions by transforming an ocean of data into streams of insight. Our clients are among Australia's top-performing companies, and they choose to partner with Capgemini for a very good reason - our exceptional people. Due to continued growth within Capgemini's Insights & Data practice, we intend to recruit a Data Engineer with relevant consulting and communication skills. If you are already working in a consultancy role, or have excellent client-facing skills gained within large organizations, we would like to discuss our consultant opportunities with you.
Roles and responsibilities:
As a PySpark Developer, you will be responsible for analyzing structured and semi-structured data using PySpark SQL queries, relevant APIs, MLib, and other tools for engineering tasks related to data. This role involves designing, building, and deploying Big Data applications, focusing on data migration, transformation, integration solutions, and operationalizing code deployment.
Requirements
- Minimum 3 years of experience in developing Big Data applications using SparkSQL, and SparkStreaming in Python.
- Demonstrated ability in database migration, transformation, and integration for Data warehousing/Lakehouse projects.
- Deep expertise in PySpark for data processing tasks including reading from external sources, data merging, enrichment, and loading into target destinations.
- Experience with deployment and operationalization, familiarity with scheduling tools like Airflow, Control-M preferred.
- Proficiency in graph algorithms, and advanced recursion techniques.
- Minimum 5 years of experience in designing, building, and deploying Python-based applications.
- Hands-on experience with XML, JSON parsing/generation, and REST API interactions.
- Good knowledge of Hadoop, Hive, Cloudera/Hortonworks Data Platform, YARN, Kafka, HBase.
- Bachelor’s degree in Engineering, Computer Science, Statistics, Econometrics, or related quantitative field with at least 5 years of experience.
- Experience managing complex large-scale Big Data environments (20Tb+).
- Strong SQL skills, adept at data exporting/importing using utilities.
- Familiarity with Cloud technologies like AWS ecosystem, Google Cloud, and BigQuery is a plus.
- Understanding of Unix/Linux and Shell Scripting.
- Data modeling experience using advanced statistical analysis, and unstructured data processing.
- Ability to build APIs for provisioning data to downstream systems using various frameworks.
- Hands-on experience with AWS S3 Filesystem operations.
- Experience with Agile delivery methodologies.
Qualifications
The following qualifications are advantageous:
- Degree in Computer Science, Information Systems, or related field.
- Minimum 8 years of industry experience.
- AWS and/or Databricks certification is highly advantageous.
- Valid visa, Australian Permanent Residency, or Australian Citizenship.
- Understanding the Financial Services Industry is beneficial.
#LI-AR1
