Senior Data Engineer

Join or sign in to find your next job

Join to apply for the Senior Data Engineer role at Troocoo

1 day ago Be among the first 25 applicants

Join to apply for the Senior Data Engineer role at Troocoo

• Design, build, and maintain scalable data pipelines and ETL processes using Azure Data Factory for data ingestion from diverse sources including on-premises databases (Authority), API-based CAMMS, and data files in CSV and JSON formats.
• Utilize Azure Data Lake Storage (Gen2) and Delta Lake for managing data in different stages (Bronze, Silver, Gold).
• Implement data processing workflows using Azure Databricks to transform raw data into processed high-quality data assets.
• Develop and optimize data models in Databricks and Power BI ensuring they meet performance and scalability requirements.
• Enable advanced analytics capabilities by providing clean, structured, and reliable data for analysis and reporting using Azure Databricks and Power BI.
• Implement and manage DevOps practices for continuous integration and continuous deployment (CI/CD) to enhance data engineering workflows through Git integration.
• Establish and enforce data standards, policies, and procedures including a data quality framework to ensure the accuracy, completeness, and consistency of data.
• Support the Data and Insights teams by providing efficient data access and processing tools on Azure.
• Monitor, support, and improve the technology landscape, including maintenance and cost savings.
• Assist the Team Leader with remediation activities identified from internal and external audits.
  
  

Key responsibilities include;

• Design, build, and maintain scalable data pipelines and ETL processes using Azure Data Factory for data ingestion from diverse sources including on-premises databases (Authority), API-based CAMMS, and data files in CSV and JSON formats.
• Utilize Azure Data Lake Storage (Gen2) and Delta Lake for managing data in different stages (Bronze, Silver, Gold).
• Implement data processing workflows using Azure Databricks to transform raw data into processed high-quality data assets.
• Develop and optimize data models in Databricks and Power BI ensuring they meet performance and scalability requirements.
• Enable advanced analytics capabilities by providing clean, structured, and reliable data for analysis and reporting using Azure Databricks and Power BI.
• Implement and manage DevOps practices for continuous integration and continuous deployment (CI/CD) to enhance data engineering workflows through Git integration.
• Establish and enforce data standards, policies, and procedures including a data quality framework to ensure the accuracy, completeness, and consistency of data.
• Support the Data and Insights teams by providing efficient data access and processing tools on Azure.
• Monitor, support, and improve the technology landscape, including maintenance and cost savings.
• Assist the Team Leader with remediation activities identified from internal and external audits.
  
  

Key requirements;

• Bachelor's degree or above in Computer Science, Data Engineering, Information Systems, or a related field.
• 5+ years of experience in data engineering, data modelling, data management, and data governance using Azure cloud services.
• Strong expertise in data architecture, ETL processes, and data modelling on Azure.
• Proficiency in programming languages such as Python and SQL with demonstratable experience in Azure Data Factory, Azure Databricks, and other Azure data services.
• Experience with DevOps practices, including CI/CD pipelines.
• Experience in ingesting data from on-premises RDBMS databases, APIs, and various formats like CSV, XLS and JSON.
• Data governance in Azure Purview and Unity Catalog.
• Strong analytical and problem-solving skills with the ability to translate business requirements into technical solutions.
• Excellent communication and collaboration skills to work effectively with cross-functional teams.

Seniority level

• Seniority levelMid-Senior level

Employment type

• Employment typeFull-time

Job function

• Job functionInformation Technology and Engineering
• IndustriesIT Services and IT Consulting, Advertising Services, and Financial Services

Referrals increase your chances of interviewing at Troocoo by 2x

Get notified about new Senior Data Engineer jobs in Sydney, New South Wales, Australia.

Sydney, New South Wales, Australia 2 weeks ago

Software Engineer - Backend (Open to remote across ANZ)

Sydney, New South Wales, Australia 3 days ago

Software Engineer - Frontend (Open to remote across ANZ)

Sydney, New South Wales, Australia 3 days ago

2025 Software Development Graduate (AWS, Auckland,NZ), (0-2 Years, Entry-Level)

Sydney, New South Wales, Australia 34 minutes ago

Software Engineer, Engineering Productivity, Android, Early CareerFrontend Software Engineer - CMS Team (Open Across ANZ)Software Engineer, Early Career, iOS, Photos

Sydney, New South Wales, Australia 4 days ago

Lane Cove West, New South Wales, Australia 1 day ago

Sydney, New South Wales, Australia 4 days ago

Software Engineer - (Java) Backend (Open to remote across ANZ)

Sydney, New South Wales, Australia 19 hours ago

Sydney, New South Wales, Australia 2 weeks ago

Sydney, New South Wales, Australia 2 weeks ago

Sydney, New South Wales, Australia 2 hours ago

Sydney, New South Wales, Australia 6 months ago

Sydney, New South Wales, Australia 2 days ago

Sydney, New South Wales, Australia 4 months ago

Sydney, New South Wales, Australia 2 weeks ago

Sydney, New South Wales, Australia A$80,000.00-A$120,000.00 3 days ago

Sydney, New South Wales, Australia 1 month ago

Sydney, New South Wales, Australia A$100,000.00-A$150,000.00 2 days ago

Sydney, New South Wales, Australia A$100,000.00-A$185,000.00 3 hours ago

Sydney, New South Wales, Australia 5 months ago

Sydney, New South Wales, Australia 2 days ago

Software Development Graduate (2025, AWS)

Sydney, New South Wales, Australia 3 days ago

Sydney, New South Wales, Australia 6 months ago

Mid-Level Software Engineer (2 Positions)

Sydney, New South Wales, Australia 1 month ago

Sydney, New South Wales, Australia 2 weeks ago

Sydney, New South Wales, Australia A$150,000.00-A$175,000.00 2 months ago

Sydney, New South Wales, Australia 2 months ago

Sydney, New South Wales, Australia 1 month ago

We're unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr

---