Advanced Data Pipeline Specialist

5 days ago

Sydney, New South Wales, Australia beBeeDataEngineer Full time
Senior Data Engineer Job Description

We are seeking a skilled Senior Data Engineer to design, build, and maintain scalable data pipelines and ETL processes using Azure Data Factory. This role requires expertise in data architecture, ETL processes, and data modeling on Azure.

The ideal candidate will have strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions. They will also possess excellent communication and collaboration skills to work effectively with cross-functional teams.

Job Responsibilities:
  • Design, build, and maintain scalable data pipelines and ETL processes using Azure Data Factory for data ingestion from diverse sources including on-premises databases, API-based systems, and various file formats.
  • Utilize Azure Data Lake Storage (Gen2) and Delta Lake for managing data in different stages (Bronze, Silver, Gold).
  • Implement data processing workflows using Azure Databricks to transform raw data into processed high-quality data assets.
  • Develop and optimize data models in Databricks and Power BI ensuring they meet performance and scalability requirements.
  • Enable advanced analytics capabilities by providing clean, structured, and reliable data for analysis and reporting using Azure Databricks and Power BI.
  • Implement and manage DevOps practices for continuous integration and continuous deployment (CI/CD) to enhance data engineering workflows through Git integration.
  • Establish and enforce data standards, policies, and procedures including a data quality framework to ensure the accuracy, completeness, and consistency of data.
  • Support the Data and Insights teams by providing efficient data access and processing tools on Azure.
  • Monitor, support, and improve the technology landscape, including maintenance and cost savings.
Requirements:
  • Bachelor's degree or above in Computer Science, Data Engineering, Information Systems, or a related field.
  • 5+ years of experience in data engineering, data modeling, data management, and data governance using Azure cloud services.
  • Strong expertise in data architecture, ETL processes, and data modeling on Azure.
  • Proficiency in programming languages such as Python and SQL with demonstratable experience in Azure Data Factory, Azure Databricks, and other Azure data services.
  • Experience with DevOps practices, including CI/CD pipelines.
  • Experience in ingesting data from on-premises RDBMS databases, APIs, and various formats like CSV, XLS, and JSON.
  • Data governance in Azure Purview and Unity Catalog.
  • Strong analytical and problem-solving skills with the ability to translate business requirements into technical solutions.
  • Excellent communication and collaboration skills to work effectively with cross-functional teams.

  • Data Pipeline Specialist

    1 day ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $100,000 - $140,000

    Data Engineering RoleWe are seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data pipelines to support data ingestion, transformation, and integration from various sources.The ideal candidate will have experience with Kafka, strong knowledge of AWS...

  • Expert Data Pipeline Specialist

    3 days ago

    Sydney, New South Wales, Australia beBeeData Full time $100,000 - $150,000

    Job Summary:We are seeking an experienced Data Pipeline Specialist to play a vital role in ensuring the quality and integrity of our data pipelines.The ideal candidate will have 5+ years of expertise in SQL, Control-M, Unix, and data warehousing.Responsibilities Include:Design and ExecutionDesign, develop, and execute test plans and test cases for ETL...

  • Data Pipeline Specialist

    7 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time

    About the RoleDrive's Data Operations team is seeking an ETL Engineer to help us achieve our vision of shaping the future of automotive in Australia. As a key member of this team, you will be responsible for designing, implementing, and maintaining efficient data pipelines that drive business value.The ideal candidate will have 5+ years of experience with...

  • Data Pipeline Specialist

    3 days ago

    Sydney, New South Wales, Australia beBeeDevopser Full time $150,000 - $170,000

    Job DescriptionAs a DevOps Engineer, you will play a vital role in our trading technology team. You will be responsible for performing daily monitoring and market pre-open system checks utilizing automation, developing new tools and utilities to help build data pipelines and studies, and becoming a subject matter expert in our in-house systems and data...

  • Data Pipeline Specialist

    9 hours ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $270,000

    Job Title: Data Engineer AssociateWe are looking for a highly skilled and motivated Data Engineer Associate to join our team. As a key member of our data engineering group, you will play a crucial role in designing, developing, and maintaining our modern data pipelines. You will be working closely with experienced engineers to develop solutions that improve...

  • Data Pipeline Specialist

    2 days ago

    Sydney, New South Wales, Australia beBeeTechnology Full time $150,000 - $170,000

    Job Description:Susquehanna is a global privately held financial trading firm with a unique approach to making business decisions every day.Their trading technology team performs support, deployment and automation tasks related to the management of data pipelines throughout the Asia Pacific region.Daily monitoring and market pre-open system checks are...

  • Data Pipeline Specialist

    2 days ago

    Sydney, New South Wales, Australia beBeeData Full time $150,000 - $200,000

    Technical Support EngineerSig-Susquehanna is a global financial trading firm that utilizes collective wisdom and relentless problem-solving to drive business decisions.Your Role:As part of our trading technology team, you will be responsible for performing support, deployment, and automation tasks related to data pipelines in the Asia Pacific region. You...

  • Data Pipeline Specialist

    4 days ago

    Sydney, New South Wales, Australia beBeeData Full time $100,000 - $150,000

    Job OpportunityWe are seeking a highly skilled and motivated Devops Engineer to join our trading technology team. This role offers a unique opportunity to work on cutting-edge data pipelines and systems, collaborating closely with developers, traders, and quant researchers.Key Responsibilities:Perform daily monitoring and market pre-open system checks...

  • Senior Data Pipeline Specialist

    2 days ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $120,000 - $180,000

    Drive is a leading automotive enthusiast brand. We're looking for a skilled professional to help us achieve our vision of shaping the future of automotive in Australia.We're a business with extensive reach across all media channels, including TV, digital, social, print, and radio. Our data operations are at the heart of this vision, and we need someone to...

  • Senior Data Pipeline Specialist

    12 hours ago

    Sydney, New South Wales, Australia beBeeDataEngineer Full time $90,000 - $120,000

    We're seeking a skilled Data Engineer to join our high-performing team. As a key member, you'll play a pivotal role in building and maintaining robust data pipelines that drive business growth.Key Responsibilities:Data Pipeline Development: Design, develop, and deploy scalable data pipelines using cutting-edge tools and technologies such as Apache Spark,...