Big Data Developer

4 days ago

Sydney, New South Wales, Australia AI Hustler Full time $104,000 - $130,878 per year

We are seeking a skilled and experienced
Big Data Engineer
to join our growing data team. You will be instrumental in building, maintaining, and optimizing the data pipelines and infrastructure that power our analytics and machine learning initiatives. This is an exciting opportunity to work with complex datasets and cutting-edge technologies in a dynamic and forward-thinking environment.

About the Role:

As a Big Data Engineer, you will be a core member of our engineering team, responsible for the end-to-end data lifecycle. You will focus on designing and implementing scalable and reliable data solutions that ensure our data is accessible, clean, and ready for use by data scientists and business analysts. Your expertise will be crucial in helping us harness the power of our data to drive innovation and strategic decision-making.

Key Responsibilities:

  • Data Pipeline Development:
    Design, build, and maintain highly scalable and efficient data pipelines (ETL/ELT) using a variety of technologies and frameworks.
  • Big Data Processing:
    Utilize distributed computing frameworks like
    Apache Spark
    to process large volumes of data from diverse sources.
  • Infrastructure & Architecture:
    Develop and maintain the data architecture, including data lakes and data warehouses, to ensure performance, security, and reliability.
  • Collaboration:
    Work closely with data scientists to understand their data requirements and ensure the data platform can support machine learning model development and deployment.
  • Data Governance & Quality:
    Implement data quality checks, validation rules, and governance policies to ensure the integrity and accuracy of our data.
  • Automation:
    Automate data workflows and implement CI/CD practices for data pipelines to ensure efficiency and reproducibility.
  • Performance Optimization:
    Troubleshoot and optimize data processes and queries to improve performance and reduce costs.

Required Skills & Qualifications:

  • Experience:
    3+ years of professional experience as a Data Engineer, Big Data Engineer, or in a similar role.
  • Programming:
    Strong proficiency in at least one of the following:
    Python
    ,
    Scala
    , or
    Java
    .
  • Big Data Ecosystem:
    Hands-on experience with
    Apache Spark
    and a solid understanding of the Hadoop ecosystem.
  • Cloud Platforms:
    Experience with at least one major cloud provider's data services (
    AWS, Azure, or GCP
    ).
  • Databases:
    Expertise in SQL and experience with both relational and NoSQL databases.
  • Data Modeling:
    Strong understanding of data modeling techniques and data warehousing concepts.
  • ETL/Orchestration:
    Experience with data orchestration tools like
    Apache Airflow
    , Azure Data Factory, or similar.
  • Communication:
    Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders.

Desirable Skills (The Edge):

  • Experience with
    streaming technologies
    like Apache Kafka or Apache Flink.
  • Familiarity with containerization and orchestration using
    Docker and Kubernetes
    .
  • Knowledge of data visualization tools such as
    Power BI
    or
    Tableau
    .
  • Prior experience with a specific cloud-based data warehouse solution like
    Snowflake
    or
    Databricks
    .

If you are a passionate Big Data Engineer with a knack for building scalable and robust data solutions, we encourage you to apply

  • Big Data Developer

    2 weeks ago

    Sydney, New South Wales, Australia Cognizant Full time

    Position SummaryWe are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines, transform...

  • Big Data Developer

    2 weeks ago

    Sydney, New South Wales, Australia Cognizant Full time

    Position SummaryWe are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines, transform...

  • Big Data Developer

    2 weeks ago

    Sydney, New South Wales, Australia Cognizant Technology Solutions Full time US$75,000 - US$85,000 per year

    Position Summary:We are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines, transform...

  • Big Data Developer

    6 days ago

    Sydney, New South Wales, Australia Cognizant Full time

    Position SummaryWe are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines, transform...

  • Big Data Developer

    3 days ago

    Sydney, New South Wales, Australia Cognizant Full time

    OverviewPosition Summary: We are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines,...

  • Big Data Developer

    3 days ago

    Sydney, New South Wales, Australia Cognizant Full time

    OverviewPosition Summary: We are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust, scalable, and high-performance big data solutions using modern big data tools and technologies. You will be responsible for developing data pipelines,...

  • Big Data Developer

    3 weeks ago

    Sydney, New South Wales, Australia Cognizant Full time

    Join to apply for the Big Data Developer role at Cognizant1 day ago Be among the first 25 applicantsJoin to apply for the Big Data Developer role at CognizantPosition SummaryWe are seeking a highly skilled and experienced Big Data Developer to join our engineering team. The ideal candidate will have strong expertise in building and maintaining robust,...

  • Big Data Developer

    1 week ago

    Sydney, New South Wales, Australia Bullit Management Services Limited Full time

    Job Description for Big Data Lead Engineer:Location: SydneyResponsibilitiesProvides technical direction and guidance to the engineering team.Ensures adherence to engineering standards and best practices.Participates in technical discussions and decision-making.Identifies and resolves technical challenges and issues.Conducts code reviews and provides feedback...

  • Big Data Developer

    2 weeks ago

    Sydney, New South Wales, Australia BULLIT MANAGEMENT SERVICES LIMITED Full time

    Job Description for Big Data Lead Engineer:Location: SydneyResponsibilitiesProvides technical direction and guidance to the engineering team.Ensures adherence to engineering standards and best practices.Participates in technical discussions and decision-making.Identifies and resolves technical challenges and issues.Conducts code reviews and provides feedback...

  • Big Data Developer

    2 weeks ago

    Sydney, New South Wales, Australia BULLIT MANAGEMENT SERVICES LIMITED Full time

    Job Description for Big Data Lead Engineer:Location: SydneyResponsibilitiesProvides technical direction and guidance to the engineering team.Ensures adherence to engineering standards and best practices.Participates in technical discussions and decision-making.Identifies and resolves technical challenges and issues.Conducts code reviews and provides feedback...