Senior Data Engineer

2 weeks ago

Council of the City of Sydney, Australia ClearCompany Full time

Due a data modernisation program moving to Fabric, we are now looking for a Senior Data Engineer to join the business.

The role of the Senior Data Engineer is responsible for designing, building, and maintaining the infrastructure that supports data storage, processing and retrieval.

You will require proven experience working with large data sets and the ability to develop data pipelines that move data from source systems to data warehouses, data lakes and other data storage and processing systems.

Qualifications

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 8+ years of experience as a Data Engineer or in a related role, with a strong focus on real-time data streaming.

- Proven experience with Microsoft Stream Analytics, Power BI, Azure Data Lake, Azure Data Factory, Azure Data Warehouse, Microsoft Fabric, Azure DevOps, SQL Server, and Python.

- Real-time Data Processing experience: Implement real-time data processing solutions using stream processing frameworks such as Kafka Streams, Apache Flink, or Apache Spark Streaming.

- Familiarity with CI/CD pipelines and distributed systems and a solid understanding of data architecture principles.

- Familiarity with event-driven architectures and microservices.

- Design and Develop Data Pipelines: Architect, develop, and maintain scalable and reliable data pipelines for real-time data processing using Microsoft Stream Analytics.

- Data Integration: Integrate various data sources (databases, APIs, file systems, etc.) with Kafka and downstream systems to ensure seamless data flow.

- Monitoring and Troubleshooting: Continuously monitor Azure data pipelines, identify issues, and troubleshoot to ensure smooth data flow and minimal downtime.

- Performance Optimization: Optimize the performance of Azure data pipelines to manage increasing data loads and maintain low latency.

- Documentation: Create and maintain comprehensive documentation for Kafka infrastructure, data pipelines, and associated processes.

- You will require full working rights for Australia.

- The client offers a hybrid working model.

Reference Number: 06800-0013322918

#J-18808-Ljbffr

  • Senior Data Engineer

    4 weeks ago

    Council of the City of Sydney, Australia Neara Group Full time

    Imagine preventing wildfires before they start, or ensuring power grids stay resilient during extreme weather. That’s the real-world impact we create every day at Neara. We build 3D physics-enabled digital twins of critical infrastructure networks that help utilities and infrastructure companies run scenarios, assess risks, and make smarter decisions that...

  • Senior Data Engineer

    2 weeks ago

    Council of the City of Sydney, Australia Atlassian Full time

    A global software company in Sydney is hiring a Senior Data Engineer to lead data solutions and drive a data-driven culture. You'll collaborate with product analysts and data scientists to build data models and pipelines, ensuring data quality and optimal storage for business needs. The ideal candidate has 5+ years of experience in data engineering, SQL...

  • Senior Data Engineer | Hybrid, Scalable Data Pipelines

    20 hours ago

    Council of the City of Sydney, Australia CoStar Group, Inc. Full time

    A global data solutions firm is seeking a Senior Data Engineer in New South Wales. The role involves building and optimizing data pipelines and transforming complex data into usable analytics products. The ideal candidate will have over 5 years of experience, strong AWS knowledge, and expertise in data engineering tools such as dbt and Snowflake. This...

  • Senior Data Platform Engineer: Build Data Pipelines

    3 weeks ago

    Council of the City of Sydney, Australia HUB24 Limited Full time

    A leading financial technology company in Sydney is seeking a Senior Engineer - Data Platforms to deliver high-quality data solutions. This role entails collaborating with product teams and using modern technologies like Databricks and GCP. Ideal candidates possess over 3 years of experience in data engineering and a degree in a relevant field. Join a...

  • Senior Manager, Data Engineering

    4 weeks ago

    Council of the City of Sydney, Australia Mable Full time

    Overview HomeMade, Mable, Leap in! and Mable Direct are entities under the Attain Healthtech Group of companies. While our companies are diverse, three similarities bind us: 1) We exist to create lasting, positive change across the entire care and support sector - impacting more lives, elevating standards and shaping the future of care; 2) We believe in...

  • Senior Data Engineer — Lead Data Platform

    4 weeks ago

    Council of the City of Sydney, Australia Singtel Group Full time

    A leading telecommunications company in Australia is seeking a Staff Data Engineer to join their Data Engineering Centre of Excellence. This senior role combines hands-on engineering with strategic leadership to shape the future of their enterprise data platform. Candidates should possess strong experience with cloud data platforms and proficiency in Python...

  • Senior Data Engineer: Cloud Data Platforms

    1 week ago

    Council of the City of Sydney, Australia UNAVAILABLE Full time

    A leading digital transformation firm in Australia seeks a Senior Associate, Data Engineering to architect innovative solutions that drive digital transformation. The ideal candidate will have expertise in cloud-native architectures, data pipelines, and a hands-on approach to collaborate with other data specialists. This role offers competitive compensation...

  • Senior AWS Data Engineer

    2 weeks ago

    Council of the City of Sydney, Australia Paxus - Technology + Digital Talent Full time

    A leading technology recruitment firm is seeking an experienced Senior Data Engineer to join their team in New South Wales. The ideal candidate will possess strong skills in data engineering and AWS Cloud to develop scalable data solutions. Key responsibilities include designing data pipelines, optimizing SQL queries, and implementing CI/CD pipelines. The...

  • Data Engineer

    7 days ago

    Council of the City of Sydney, Australia Stake Australia Full time

    About this role As a Data Engineer at Stake you will design, build, and maintain a robust and scalable data infrastructure to support the Stake data ecosystem, ensuring data quality, reliability, and accessibility. In collaboration with technical and non-technical peers you will define and implement best practices for data infrastructure and...

  • Senior Data Engineer: Real-Time Pipelines

    5 days ago

    Council of the City of Sydney, Australia Neoen Full time

    A leading energy management firm in Australia is looking for a Senior Data Engineer to manage data storage, access, and ingestion for energy market applications. Responsibilities include maintaining critical databases, developing efficient data pipelines, and mentoring junior team members. Ideal candidates should have solid experience with SQL Server and...