Big Data Engineer for Large-Scale Data Solutions

6 days ago

Sydney, New South Wales, Australia Luxoft Full time
Job Summary

Luxoft is seeking a highly skilled Big Data Engineer to design, develop, and maintain large-scale data solutions using PySpark. As a key member of our team, you will work closely with data scientists, data analysts, and software engineers to drive robust, data-driven insights for business decisions.

About the Role

We are looking for an experienced engineer who can optimize ETL pipelines for performance and scalability. You will work with large volumes of structured and unstructured data, transforming it to meet business needs. Your expertise in integrating data from multiple sources into the data platform will ensure data integrity and quality.

In this role, you will collaborate with cross-functional teams to understand data requirements and translate them into efficient data workflows. You will implement best practices for data governance, monitoring, and data security. Additionally, you will debug and troubleshoot issues across ETL pipelines and data workflows.

Key Responsibilities
  • Design, develop, and maintain ETL pipelines using PySpark, optimizing for performance and scalability.
  • Work with large volumes of structured and unstructured data, transforming it to meet business needs.
  • Integrate data from multiple sources into the data platform, ensuring data integrity and quality.
  • Collaborate with cross-functional teams to understand data requirements and translate them into efficient data workflows.
  • Implement best practices for data governance, monitoring, and data security.
Required Skills and Qualifications

To be successful in this role, you will need:

  • Proficiency in PySpark: In-depth experience with PySpark for data processing and transformation tasks.
  • SQL Knowledge: Strong command of SQL for querying and processing data.
  • Data Warehousing Concepts: Familiarity with data warehousing, data lakes, and data integration principles.
  • Cloud Platforms: Experience with cloud environments like AWS, GCP, or Azure for data storage and processing.
  • Big Data Technologies: Hands-on experience with Hadoop and Spark ecosystem (Spark SQL, Spark Streaming).
  • Data Modeling: Experience in designing and implementing efficient data models.
  • Python Programming: Strong Python skills, particularly in data manipulation and analysis.
Estimate Salary: $120,000 - $180,000 per year

  • Data Property Solutions Engineer

    1 day ago

    Sydney, New South Wales, Australia Data-Property Data Solns Shared Full time

    We are seeking a highly skilled Solutions Data Engineer to join our team at Data-Property Data Solns Shared. As a Senior member of our organization, you will play a critical role in shaping the future of property data solutions.About UsWe shine a light on all things property, aiming to simplify the property journey for all involved. Our business is motivated...

  • Senior Software Engineer

    3 weeks ago

    Sydney, New South Wales, Australia Dicker Data Full time

    Job Description:As a senior software engineer in data analysis, you will be responsible for developing and maintaining large-scale data processing systems. Your primary focus will be on designing and implementing data pipelines using Python, ensuring data quality and integrity. You will work closely with cross-functional teams to deliver high-quality...

  • Senior Data Engineer: Big Data Platform Expert

    2 days ago

    Sydney, New South Wales, Australia Commonwealth Bank of Australia Full time

    We are seeking an experienced Senior Data Engineer to join our team in the Chief Data Office. This is a critical role that will drive the development and implementation of cutting-edge data operations solutions using ETL tools and big data platforms.As a Senior Data Engineer, you will work closely with internal teams and stakeholders to design, build, and...

  • Data Engineer

    3 days ago

    Sydney, New South Wales, Australia Data - AI & Data Science Full time

    About UsWe shine a light on the property market, simplifying the journey for all involved. Our expertise and exclusive data drive change. At Domain Group, our data department leads the way in driving value across the organisation.The RoleAs a Senior Data Engineer, you'll play a key role in building data pipelines that power Australia's property market....

  • Big Data Solutions Architect

    3 weeks ago

    Sydney, New South Wales, Australia Zone IT Solutions Full time

    About the RoleWe are seeking a highly skilled Big Data Solution Engineer to join our team at Zone IT Solutions.Key ResponsibilitiesDesign and implement scalable big data solutions to support our clients' analytical needs.Develop and maintain effective data models and data ingestion processes.Collaborate with cross-functional teams to deliver big data...

  • Big Data Solutions Developer

    3 weeks ago

    Sydney, New South Wales, Australia Cloudera Full time

    Job Title: Solutions ArchitectAbout the Role:We are seeking an experienced Solutions Architect to join our ANZ Professional Services team at Cloudera. As a Solutions Architect, you will have the opportunity to design and implement large-scale Big Data solutions using the Cloudera Data Platform and Cloudera Dataflow.Key Responsibilities:Work directly with...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Whizdom Recruitment Full time

    Job Title: Data Engineer - Big Data SolutionsCompany Overview: Whizdom Recruitment is a leading recruitment agency specialising in IT and data science professionals. We're currently looking for a highly skilled Data Engineer to join our team.Salary: $120,000 - $150,000 per annum, depending on experienceJob Description: As a Data Engineer, you will be...

  • Data Engineer for Big Data Platforms

    4 days ago

    Sydney, New South Wales, Australia Whizdom Recruitment Full time

    Job Overview:We are seeking an experienced Data Engineer to join our team at Whizdom Recruitment. As a key member of our big data platform, you will be responsible for building and supporting a robust and scalable data infrastructure.About the Role:The ideal candidate will have a strong background in data engineering, with experience in designing and...

  • Big Data Solutions Architect

    4 weeks ago

    Sydney, New South Wales, Australia Zone IT Solutions Full time

    We are seeking a highly skilled Big Data Solutions Architect to design and implement scalable big data solutions that support our clients' analytical needs.Key ResponsibilitiesDesign and implement big data architectures using Hadoop ecosystem components and relevant data storage solutions.Develop and maintain data processing frameworks and tools like Apache...

  • Big Data Solutions Architect

    4 weeks ago

    Sydney, New South Wales, Australia Zone IT Solutions Full time

    We are seeking a highly skilled Big Data Solutions Architect to join our team at Zone IT Solutions. This role involves designing and implementing scalable big data solutions to support our clients' analytical needs.Key Responsibilities:Design and implement big data solutions using Hadoop ecosystem components and relevant data storage solutions.Develop and...

  • Big Data Engineer

    5 days ago

    Sydney, New South Wales, Australia Zone IT Solutions Full time

    We are seeking a highly skilled Big Data Engineer to join our team in Melbourne, Sydney, or India.Job Summary:Zone IT Solutions is a leading recruitment company specializing in Digital, ERP, and larger IT Services. We require an experienced Big Data Engineer to design and implement scalable big data solutions for our clients' analytical needs.About the...

  • Big Data Solution Architect

    4 days ago

    Sydney, New South Wales, Australia Zone IT Solutions Full time

    Job Title: Big Data Solution ArchitectAbout UsZone IT Solutions is a leading recruitment company based in Australia, specializing in digital, ERP, and larger IT services. We provide flexible, efficient, and collaborative solutions to organizations seeking top-notch IT expertise.Salary RangeThe estimated salary for this position is between $120,000 - $180,000...

  • Big Data Solutions Architect

    4 days ago

    Sydney, New South Wales, Australia Commonwealth Bank of Australia Full time

    About Commonwealth Bank of AustraliaWe are one of Australia's most advanced big data operations teams, responsible for CommBank's key analytics capabilities and world-leading capabilities for analytics, information management and decisioning.Our team works across the Group Data Warehouse, Teradata/mainframe, Big Data Platform, Pega Decisioning, SAS, R,...

  • Big Data Solutions Architect

    4 weeks ago

    Sydney, New South Wales, Australia Cloudera Full time

    At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world's...

  • Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia The Drive Group Full time

    Job Title: Data Engineer - Big Data SpecialistA leading Australian eCommerce company is growing its data capability and is looking to hire a Data Engineer - Big Data Specialist.As the new Data Engineer - Big Data Specialist, you will be responsible for building data services that fulfill the data and reporting needs of the business as they continue to grow...

  • Senior Data Engineer

    3 weeks ago

    Sydney, New South Wales, Australia Experis Full time

    Senior Data Engineer - Big DataExperis is representing a prestigious financial services organisation in recruiting a Senior Data Engineer - Big Data to lead the design, development, and delivery of future-ready data solutions. As a technical expert, you will work closely with cross-functional teams to provide cutting-edge data solutions while mentoring...

  • Data Engineer

    1 week ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    About UsCDC Data Centres is a leading operator of sovereign, secure, world-class data centre facilities in Australia. Established in 2007, we provide diversified operations with secure data centre configurations that support co-location, containerised and hyperscale compute environments.Job DescriptionWe are seeking a highly skilled Senior Data Engineer to...

  • Data Engineer

    1 day ago

    Sydney, New South Wales, Australia Data - AI & Data Science Full time

    Drive Business Value with Data Engineering at Domain GroupWe are seeking a skilled Data Engineer to join our Data Decision Science team in Sydney. As a key member of our data department, you will play a critical role in building scalable data pipelines that power Australia's property market.About the RoleThis is an exciting opportunity to work with...

  • Big Data Solutions Architect

    3 days ago

    Sydney, New South Wales, Australia Whizdom Recruitment Full time

    Seeking a seasoned Data Solutions Architect to drive advanced data solutions across industries, leading technical aspects of data projects and guiding data engineers and testers. This role involves designing, developing, and supporting future-ready data solutions for customers across industry verticals.The ideal candidate has 5+ years in solution design or...

  • Senior Data Engineer

    4 weeks ago

    Sydney, New South Wales, Australia Cdc Data Centres Full time

    About UsCDC Data Centres is a leading operator of sovereign, secure, world-class data centre facilities in Australia. Our mission is to attract, retain, and grow the world's best data centre team.About the RoleWe are seeking a Senior Data Engineer to design, implement, and support telemetry systems and reporting solutions within our IT environments. The...