Data Engineer

At TL Consulting, we simplify modern complexity with advanced, sustainable engineering. As a proudly Australian owned consultancy with over 15 years of experience, we empower businesses

to innovate and achieve measurable outcomes through scalable, cloud-native solutions. Our 50+ consultants operate across key sectors including banking, financial services & insurance (BFSI), government, retail, education, and entertainment. This cross-industry expertise allows us to deliver customer-centric solutions that combine deep technical capability with a clear understanding of each client's strategic vision. Our approach leverages proven, scalable blueprints tailored to your business direction—enabling accelerated outcomes, maximized ROI, and sustainable long-term adoption.

We maintain strategic partnerships with organizations that share our vision for a cloud-native future. Through our collaborations with Microsoft, GitHub, and Databricks, we ensure our capabilities stay aligned with evolving best practices. These partnerships also provide access to innovation programs, co-funding opportunities, and dedicated support—helping us accelerate time-to-value for our clients.

About role

We are looking for a highly skilled Data Enginee
r
to join our team. This role is critical in designing, implementing, and optimising
data pipelines
for a large-scale data migration initiative. The ideal candidate must have
strong expertise in Fabric component (Dataflows, Pipelines, Lakehouse and warehouse) Azure Data Factory (ADF)
,
ETL processes, Python coding,
SQL Server
, and
Azure Storage (Blob, ADLS Gen2)
.

Key Responsibilities:

1. Data Pipeline Development & Migration

• Design, develop, and manage Azure Data Factory (ADF) pipelines to facilitate end-to-end data migration.
• Implement full load and incremental load strategies for migrating data from various sources to the target system using Fabric components such as Dataflows, Pipelines, Lakehouse and warehouse.
• Handle pipeline scheduling, monitoring, and execution ensuring reliability and minimal downtime.
• Optimise data pipelines for performance and scalability.

2. Data Transformation & Cleansing

• Ensure data integrity, accuracy, and consistency through transformation and cleansing techniques using Fabric Notebook with Python code.
• Develop ETL logic to map source-to-target transformations efficiently.
• Implement data validation checks to identify and handle discrepancies during migration.

3. Exception Handling & Troubleshooting

• Implement robust error-handling mechanisms in Azure Data Factory pipelines.
• Identify, diagnose, and resolve issues related to data processing, transformation failures, or connectivity problems.
• Maintain detailed logs and proactively monitor data flow issues.

4. SQL Development & Database Management on Azure

• Develop complex SQL queries, stored procedures, functions, and scripts for data extraction, transformation, and loading (ETL).
• Work with Azure SQL Server databases to design and optimise data models for efficient migration.
• Perform data profiling, indexing, and partitioning for enhanced query performance.
• Support data warehouse development and maintenance.

5. Azure Storage & Cloud Integration

• Work extensively with Azure Storage solutions (Blob & ADLS Gen2) to manage and process large datasets.
• Ensure secure and optimised data transfer between environments.
• Manage access permissions and security compliance for Azure data storage.

6. Testing & Validation

• Conduct pre-migration and post-migration data validation to ensure completeness and accuracy.
• Create test cases and scripts to verify data integrity across multiple environments.
• Work with QA teams and business users to validate data consistency after migration.

7. Collaboration & Documentation

• Work closely with business analysts, solution architects, and other stakeholders to understand data migration requirements.
• Document ETL processes, Python coding data mappings, pipeline configurations, and troubleshooting steps for knowledge sharing.
• Provide technical support and guidance to the team regarding data engineering best practices.

Required Qualifications & Skills:

1. Experience & Background

• 5-6 years
  of hands-on experience in
  data engineering, data migration, and ETL development
  .
• Strong background in
  Azure Data Factory (ADF)
  with expertise in:
• Pipeline planning, designing, execution, and monitoring
• Handling full and incremental loads
• Exception handling and performance tuning
• Data transformation and cleansing
• Experience with
  SQL Server
  , including
  database design, writing complex stored procedures, and performance optimisation
  .

2. Technical Skills

• Strong expertise in
  Azure Data Factory (ADF)
  and its integration with other Azure services.
• Proficiency in
  SQL Server
  and
  data warehousing concepts
  .
• Hands-on experience in
  Azure Storage solutions (Blob, ADLS Gen2)
  for handling structured and unstructured data.
• Good understanding of
  ETL frameworks and data transformation techniques
  .
• Experience with
  pipeline scheduling, exception handling, and monitoring
  .
• Knowledge of
  Power BI or other data visualisation tools
  is an added advantage.
• Familiarity with
  Microsoft Fabric
  is a plus.

3. Problem-Solving & Analytical Thinking

• Strong problem-solving skills to
  troubleshoot migration challenges
  .
• Ability to analyse large datasets and identify inconsistencies or performance bottlenecks.
• Experience in
  optimising SQL queries and ADF pipelines
  for better performance.

4. Collaboration & Communication

• Ability to communicate effectively with
  technical and non-technical stakeholders
  .
• Experience working in
  cross-functional teams
  to align data migration activities with business goals.
• Strong documentation skills to maintain clear and comprehensive project records.

Preferred Qualifications (Good to Have):

• Experience with
  Microsoft Fabric
  for data integration and processing.
• Proficiency in
  Power BI or similar BI tools
  for reporting and visualisation.
• Knowledge of
  data governance and compliance standards
  in cloud environments.
• Familiarity with
  other Azure cloud services
  related to data engineering.

This is a Melbourne based, contract opportunity. No agencies please. Candidates must have valid visa with full work rights in Australia. Visas will not be transferred or sponsored.

TL Consulting Group understands the importance of fostering a diverse and inclusive workplace. We are committed to ensuring that everyone, no matter their gender, identity, sexual orientation, race, ethnicity, religion, disability, or age, has equal access to opportunities. We warmly welcome applications from all individuals, including Aboriginal and Torres Strait Islander peoples, members of the LGBTQI+ community, and individuals with disabilities.