Senior Kafka Streaming Engineer

3 days ago

Greater Sydney Area, Australia Excolo. Full time $120,000 - $180,000 per year

We're partnering with a leading financial services organisation seeking an experienced Senior Kafka Streaming Engineer to design, implement and support real-time data streaming solutions across a hybrid environment. You'll play a key role in building scalable pipelines using Kafka, SQL Server CDC, Terraform and Databricks, integrating on-premises and AWS cloud systems to deliver reliable data flow for analytics and reporting.

This is a hands-on engineering role focused on enabling real-time data movement and ensuring performance, reliability and automation across the streaming platform.

Key Responsibilities

  • Design, configure and manage Kafka-based data streaming pipelines for financial systems
  • Implement SQL Server Change Data Capture (CDC) processes and stream data to Kafka topics
  • Develop and manage source and sink connectors, particularly for SQL Server and Databricks
  • Use Terraform to provision and manage Kafka infrastructure including topics, connectors, ACLs and Schema Registry configurations
  • Automate deployments and topic management using CI/CD pipelines
  • Collaborate with DevOps, data engineering and cloud teams to support end-to-end data delivery and performance optimisation
  • Enhance observability, troubleshoot issues and maintain operational documentation

Required Experience

  • Strong hands-on experience with Kafka ecosystem components
  • Proven track record implementing Kafka streaming pipelines in production environments
  • Experience streaming data from SQL Server using CDC
  • Skilled in Terraform and infrastructure-as-code for Kafka provisioning and configuration
  • Working knowledge of CI/CD pipeline automation
  • Proficiency in Python, PySpark or Scala for scripting, automation and integration tasks
  • Familiarity with Databricks as a data processing and analytics platform
  • Experience with AWS cloud services in hybrid or cloud-native architectures

Preferred Skills

  • Experience with dbt for downstream data transformation
  • Understanding of Kafka security configurations (SSL, SASL, RBAC)
  • Exposure to hybrid or cloud-native Kafka deployments
  • Background in financial services or real-time data environments

This is an exciting opportunity for a skilled Senior Kafka Streaming Engineer to join a high-performing team and make an immediate impact on a large-scale streaming data initiative.

  • Senior Kafka DataOps Engineer

    2 days ago

    Greater Sydney Area, Australia Excolo. Full time $120,000 - $180,000 per year

    We're working with a globally recognised financial services organisation that's building a distributed, scalable and reliable data streaming platform. They're looking for a Senior Kafka DataOps Engineer to design, build and manage high-performance streaming infrastructure that underpins mission-critical systems.You will provision, configure and maintain...

  • Kafka Engineer

    2 weeks ago

    Sydney, New South Wales, Australia Optiver Full time $120,000 - $180,000 per year

    Who We AreOptiver is a global market maker founded in Amsterdam, with offices in London, Chicago, Austin, New York, Sydney, Shanghai, Hong Kong, Singapore, Taipei and Mumbai. Established in 1986, today we are a leading liquidity provider, with close to 2,000 employees in offices around the world, united in our commitment to improve the market through...

  • Power Query and Kafka Consultant

    1 week ago

    Sydney, Australia Delivery Centric Technologies Full time

    We are currently looking for **Power Query AND Kafka Consultant **who are based out of **Sydney**, to help our clients deliver solutions. **Role Title: Power Query AND Kafka Consultant** **Location: Sydney** **Employment Type**:Full Time** **Responsibilities and Duties** We are seeking a skilled **Power BI Developer with expertise in integrating Power...

  • Kafka Admin

    1 week ago

    Sydney, Australia Delivery Centric Technologies Full time

    **Job Title: Kafka Admin** **Location: Sydney, NSW** **Employment Type: Permanent** **(Note: Hiring PRs and Citizens only)** **Requirements**: - Proven experience as a **Kafka Administrator** in large-scale, production environments. - Strong understanding of Kafka architecture, internals, and best practices. - Proficient in **scripting languages (e.g.,...

  • Senior Platform Engineer

    2 weeks ago

    Greater Sydney Area, Australia Cognizant Full time $100,000 per year

    What makes Cognizant a unique place to work? The combination of rapid growth and an international and innovative environment This is creating many opportunities for people like YOU — people with an entrepreneurial spirit who want to make a difference in this world.At Cognizant, together with your colleagues from all around the world, you will collaborate...

  • Senior Data Engineer

    2 weeks ago

    Sydney, New South Wales, Australia AI Hustler Full time $120,000 - $180,000 per year

    We're partnering with a rapidly evolving leader in theLogistics and Supply Chainsector, dedicated tooptimisingcomplex global operations and improving operational efficiency through data. For the right candidate with the necessary skills and experience, we are pleased to offer482 visa sponsorship.This client requires aSenior Data Engineerto act as a technical...

  • Senior Integration Engineer

    6 days ago

    Greater Sydney Area, Australia Excolo. Full time $120,000 - $180,000 per year

    We're looking for a Senior Integration Engineer to design and deliver secure scalable integrations across our fintech platforms. You'll work primarily with Azure integration services and a leading iPaaS platform (MuleSoft, Boomi or Workato) to build APIs and workflows that keep data flowing across core systems.What you'll doBuild and maintain APIs and...

  • Senior Python Backend Engineer

    3 days ago

    Greater Sydney Area, Australia Themelia Full time $120,000 - $180,000 per year

    Senior Python Backend Engineer (Performance & Parallel Execution)Company DescriptionThemelia is a crypto index and analytics platform entering a performance and scaling phase. We're seeking a senior Python engineer with expertise in computational optimisation and concurrent systems to improve index calculation performance and parallelise exchange trade...

  • Senior Software Engineer

    3 days ago

    Greater Perth Area, Australia Synechron Full time $120,000 - $180,000 per year

    We are seeking two experienced Senior Software Engineers to join the client's Tech team. In this role, you will work on developing and maintaining enterprise-grade solutions, utilizing a broad spectrum of tools and technologies. Your expertise will contribute to the design, development, and deployment of scalable and secure applications, supporting their...

  • Senior Data Engineer

    3 days ago

    Sydney, New South Wales, Australia Pathway Search Full time $100,000 - $175,000 per year

    Senior Data Engineer (Snowflake)Salary: up to $175k + Super Sydney - Remote options available  Help shape the future of retail media by building scalable, high-performance data solutions that power innovation and growth. Key Responsibilities Design, develop, and optimize scalable ETL/ELT data pipelines (batch & streaming).Architect and manage robust data...