Data Pipeline Developer

7 days ago

Melbourne, Victoria, Australia beBeeDataEngineer Full time $100,000 - $150,000
Key Player in Data Pipeline Development

The ideal candidate will possess a strong expertise in crafting and refining data pipelines within the Azure ecosystem, with a focus on designing, developing, and managing robust data pipelines using Azure Data Factory (ADF) and Microsoft Fabric components.

Responsibilities will include designing and developing data pipelines that ensure high data quality and integrity through meticulous transformation, cleansing, and validation using Python and PySpark.

  • Implement and optimize full and incremental data load strategies for large-scale data migration projects.
  • Develop and manage complex SQL queries, stored procedures, and scripts for data extraction and loading.
  • Collaborate extensively with cross-functional teams to align data migration activities with business objectives.

Qualifications and skills required include 5-6 years of hands-on experience in data engineering, data migration, and ETL development.

Strong expertise is needed in Azure Data Factory (ADF), including pipeline design, performance tuning, and exception handling.

  • Proficiency in SQL and experience with SQL Server database design and optimization are necessary.
  • Experience with Azure Storage solutions (Blob & ADLS Gen2) is also required.
  • Knowledge of Python and PySpark for data transformation and cleansing is a must.
  • Familiarity with Microsoft Fabric components (Dataflows, Pipelines, Lakehouse, and Warehouse) is highly desirable.

  • Cloud Data Pipeline Developer

    2 weeks ago

    Melbourne, Victoria, Australia beBeeData Full time $122,736 - $132,847

    Data Engineering OpportunityWe are seeking an experienced Data Engineer to join our team. This is a permanent opportunity with a strong focus on cloud-based pipeline development.Design, build, and optimize scalable AWS Glue (PySpark) pipelines for structured and semi-structured data sources using industry-standard practices and tools.Develop clear and...

  • Key Data Pipeline Developer

    2 weeks ago

    Melbourne, Victoria, Australia beBeeDataEngineer Full time $120,000 - $140,000

    Build Scalable Data Pipelines as a Key Team MemberThe role of Data Engineer is pivotal in our organization, requiring collaboration with stakeholders to design and develop data pipelines that meet business needs. This involves working closely with the technology team to build robust end-to-end solutions.Responsibilities:Develop and maintain high-quality data...

  • Data Pipeline Developer Intern

    5 days ago

    Melbourne, Victoria, Australia beBeeDataEngineer Full time $4,080 - $6,720

    Job OverviewThis internship offers hands-on experience in developing and maintaining data pipelines, integrating disparate data sources, and ensuring high-quality data for business operations.Key Responsibilities:Assist in designing and implementing ETL/ELT data pipelines.Integrate data from multiple sources into centralized warehouses or data lakes.Clean,...

  • Data Systems Pipeline Developer

    4 days ago

    Melbourne, Victoria, Australia Bebeeengineer Full time

    About the Role">">The Data Acquisition Team is responsible for ingesting, validating and orchestrating raw data from internal systems and third-party providers.">">As a Software Engineer you'll design and build robust scalable and extensible pipelines that feed the entire data ecosystem.">">You'll work across multiple data domains ingestion patterns batch...

  • Data Systems Pipeline Developer

    6 days ago

    Melbourne, Victoria, Australia beBeeEngineer Full time $120,000 - $180,000

    About the Role">">The Data Acquisition Team is responsible for ingesting, validating and orchestrating raw data from internal systems and third-party providers.">">As a Software Engineer you'll design and build robust scalable and extensible pipelines that feed the entire data ecosystem.">">You'll work across multiple data domains ingestion patterns batch...

  • Data Pipeline Engineer

    2 weeks ago

    Melbourne, Victoria, Australia beBeeData Full time $150,000 - $200,000

    About the Role">The Data Acquisition Team is responsible for ingesting, validating and orchestrating raw data from internal systems and third-party providers. As a Software Developer, you'll play a key role in designing robust, scalable pipelines that feed the entire data ecosystem.">The team builds the foundation of WEX's data platform transformation,...

  • Building Efficient Data Pipelines

    1 week ago

    Melbourne, Victoria, Australia beBeeData Full time $160,000 - $200,000

    Key Data Engineering RoleCraft and implement high-performance data pipelines that support our entire data ecosystem as a skilled Software Development Data Engineer.Pipeline Architecture OverviewBuild efficient data acquisition pipelines integrating with various systems across the industry, leveraging software development principles and system architecture...

  • Data Pipeline Specialist

    2 weeks ago

    Melbourne, Victoria, Australia beBeeETLDeveloper Full time $125,000 - $175,000

    Job Overview">The ETL Developer is responsible for crafting and maintaining robust data pipelines that extract, transform, and load data from diverse sources into centralized repositories.">Key Responsibilities:">">Design, develop, test, and maintain data pipelines to extract, transform, and load data from various sources into centralized...

  • Data Pipelining Specialist

    2 weeks ago

    Melbourne, Victoria, Australia beBeeDeveloper Full time $100,000 - $140,000

    Job OverviewAs a ETL Developer, you will be responsible for designing and implementing robust data pipelines to extract, transform, and load data from various sources into data warehouses.The ideal candidate will have experience in Python programming and strong analytical skills. They will be able to write clean, efficient, and logical code and provide...

  • Data Pipeline Architect

    2 weeks ago

    Melbourne, Victoria, Australia beBeeDataEngineer Full time $122,736 - $132,847

    Key Data Engineer RoleThe Australian BioCommons team is seeking a highly skilled data engineer to contribute to the establishment of OMIX3, Australia's first integrated, multi-omic analysis platform.Primary Responsibilities:Pipeline Development: Design and implement production-grade tools for automated data flows, including archiving, cataloguing, and...