Urgent Search: Senior Data Engineer

3 weeks ago


Newcastle, Australia ClearCompany Full time

Senior Data Engineer: Firewall Automation Program

- Contract: 6-12 months (Pty Ltd only)

- $1,150-$1,200/day + GST

- Sydney based (Hybrid)

- Urgent start - interviewing immediately

About the role

Build MS SQL ETL pipelines that consolidate 5 million network flows down to 5,000-10,000 automated firewall rules for a tier-1 bank's CoreMod migration. You'll enrich NetFlow data with CMDB/DNS/VMware metadata, produce JSON/CSV deployment feeds for APIC/CSM, and optimise performance at scale. This role suits a data engineer who thrives on genuinely interesting technical problems in collaborative mob programming environments.

About the engagement

Major infrastructure transformation migrating 400 applications across ~3,000 servers. Three-person specialist team (you, Python dev, network engineer) automating firewall implementation across Legacy and CoreMod infrastructure. Fast-tracked delivery with high autonomy and creative problem-solving.

What you'll own

Data Pipeline: Consolidate 5M network flows stored in MS SQL into contextualised, validated firewall rules using high-performance T-SQL ETL jobs with deduplication, enrichment and sliding-window aggregations.

Deployment Feeds: Produce controller-ready JSON/CSV exports conforming to CSM import formats and APIC object model payloads that enable the Python dev to automate rule implementation via Cisco APIs.

Orchestration: Create Tidal job definitions, retries, alerts and exception queues. Implement lineage tracking from aggregated tuples back to raw flows for audit purposes.

This is collaborative data engineering - mob programming setup where all three specialists work in lockstep. You'll optimise partitioning, indexing and query plans for very large NetFlow tables while navigating access problems and data challenges with a proactive mindset.

What we need

- Must have: Expert MS SQL Server (T-SQL, indexing, partitioning, window functions, query tuning), high-performance ETL pipelines for large-scale datasets (5M+ rows), controller-ready export formats (JSON/CSV) for network automation platforms, Tidal or similar orchestration tools, data engineering mindset (NOT DBA), collaborative team player, Pty Ltd contractor.

Nice to have

- Dask/Spark experience

- Power BI

- Understanding of network flows/NetFlow data

- Migration project experience

Won't work

- Pure DBA backgrounds

- Expecting cloud migration work

- Needing extensive banking experience

The opportunity

Real infrastructure impact enabling automated firewall deployment at enterprise scale. Strong likelihood of 12-month engagement.

#J-18808-Ljbffr


  • Clinic Manager

    1 week ago


    Newcastle West, Australia Heal Specialist Urgent Care Full time

    **At Heal, we’re on a mission to revolutionise the way urgent medical care is delivered in Australia by**: - Making healthcare easy - Delivering extraordinary care - Caring for people, not patients - Empowering people by growing health literacy We are building a team of clinical and support excellence, to challenge the status quo in healthcare and help...


  • Newcastle, Australia ClearCompany Full time

    Overview It's in their Parramatta office where they're looking for an exceptional Senior Structural Engineer to lead a team of 5 and manage projects. Responsibilities - Managing the delivery of all design aspects of the project on time, on budget and to standards - Design and Project Management of medium to high rise building projects including...


  • Newcastle, Australia Domain Group Full time

    Data Quality Officer Job Description Company Overview: CoStar Group (NASDAQ: CSGP) is a leading global provider of commercial and residential real estate information, analytics, and online marketplaces. Included in the S&P 500 Index and the NASDAQ 100, CoStar Group is on a mission to digitize the world’s real estate, empowering all people to discover...


  • Newcastle, Australia John Holland Pty Ltd Full time

    Job Number: 21098 Location: Mascot, NSW, AU, 2020 At John Holland, our purpose is simple – we transform lives with everything we do. We’ve always known that infrastructure is about people — our customers, our employees, and the communities in which we work every day. That’s our difference. Deep experience and capability with a genuine care about...


  • Newcastle, Australia ClearCompany Full time

    Overview Senior Business Analyst - Merchant Payments Location: Sydney (Hybrid) Contract Type: Day Rate Contract Rate: Competitive daily rate (based on experience) Eligibility: Sydney-based candidates only We're seeking an experienced Senior Business Analyst to join a leading Australian financial institution on a Merchant Payments program. Key...


  • Newcastle, New South Wales, Australia VIS Global Pty Ltd Full time $120,000 - $180,000 per year

    Job Title: Senior Network EngineerDesign, deploy, and manage LAN, WAN, WLAN, and cloud network infrastructures.Monitor network performance and troubleshoot complex issues related to connectivity, latency, throughput, and system integration.Plan and execute upgrades, maintenance, and expansion of existing infrastructure.Implement and enforce security...


  • Newcastle, New South Wales, Australia Advanced Navigation Full time $120,000 - $180,000 per year

    Advanced Navigation - It's In The NameAdvanced Navigation is a global leader in navigation and autonomous systems. By leveraging capabilities in software-enhanced hardware, every solution delivers unrivaled capabilities and exceptional performance across land, air, sea and space applications where GPS is unreliable.Made possible with extensive research,...


  • Newcastle, Australia Notre Dame Full time

    Research Assistant, School of Medicine, Sydney Campus, Part‑time opportunity (0.6), fixed‑term contract for 12 months - Friendly and collaborative work environment - Rewarding opportunity offering career development - Level 4 from $80,208 + 14.50% super (pro‑rata) About the university The University of Notre Dame Australia is a private Catholic...


  • Newcastle, Australia Alliance Automation Pty Full time

    Join Alliance Automation: Where Excellence Meets Opportunity! Transforming Australian Industry with Secure Digital Solutions, Smart Automation, and Electrical Engineering. We believe that our people are our most valuable asset. With a strong emphasis on integrity, trust, and honesty, we incorporate business ethics into everything we do. Join us and become...


  • Newcastle, Australia ClearCompany Full time

    Feel empowered in this Innovative team with AI and data analytics in its DNA. You'll drive results that combines cutting edge advanced analytics and automated AI with an integrated suite of leading tools and technologies. We have an exciting opportunity for a Senior Data Analyst to join a Data Analytics practice within our Financial Services client where...