Senior Data Engineer

3 days ago

Brisbane, Queensland, Australia Celebal Technologies Full time $120,000 - $200,000 per year

Celebal is looking for Data Engineer/Senior Data Engineer.

Company Profile
- Celebal Technologies a Premier Microsoft Azure partner and have been recently funded by Norwest Venture Capital. Celebal Technologies is a premier software services company in the field of Data Science, Big Data, Enterprise Cloud & Automation. Established in 2016, in this short span of time we grew to headcount of 2200+ We help in achieving a competitive advantage with intelligent data solutions, built using cutting-edge technology.

Key Responsibilities:


• Lead the implementation of data ingestion pipelines, ensuring optimized data flow and transformation logic.


• Oversee and guide the migration of DBT ETL scripts, rewriting or optimizing SQL and PySpark code for Databricks.


• Expertise in DBT, SQL Server, Synapse (T-SQL, performance tuning) and data migration to cloud platforms.


• In-depth knowledge of Databricks, Spark, Delta Lake, and integration with Azure services.


• Design and implement reusable frameworks for data transformations, logging, and error handling in Databricks.


• Strong understanding of cloud-based architecture for data lakes, data warehouses, and scalability.


• Expertise in Azure Data Factory (ADF), Synapse Analytics, and Azure Blob Storage integration with Databricks.


• Experience in implementing and managing Medallion architecture (Bronze, Silver, Gold) in Databricks.


• Optimize Spark configurations, cluster sizing, and execution strategies for performance improvements.


• Conduct code reviews to ensure adherence to best practices, performance optimizations, and security guidelines.


• Debug and troubleshoot issues related to data ingestion, transformation, and job failures in Databricks.


• Collaborate with data engineers and analysts to ensure transformed data meets business requirements and reporting needs.


• Implement robust logging and monitoring using tools such as Datadog, Prometheus, or Azure Monitor.


• Work closely with cloud and DevOps teams to enable infrastructure as code (IaC) for Databricks workspace management.


• Provide guidance on version control strategies, branching, and release management for data pipeline


• Act as a bridge between the Solution Architect and the development team, ensuring seamless implementation.


• Ensure data quality by implementing validation checks, data reconciliation, and automated data profiling.


• Strong collaboration and communication skills for leading cross-functional teams and reporting to stakeholders.


• Document all technical aspects, including ETL flows, DevOps workflows, and migration steps for knowledge sharing.


• Mentor junior engineers, conduct knowledge-sharing sessions, and enforce coding standards across the team.

Technical Skills:


• Expertise in Databricks, Spark, PySpark, Delta Lake, and SQL


• Strong hands-on experience with DBT, SQL Server, and T-SQL performance tuning


• Proficiency in Azure cloud services (Data Factory, Synapse, Blob Storage, etc.)


• CI/CD implementation for data pipelines using Azure DevOps, Terraform, or GitHub Actions


• Experience with Medallion architecture and scalable data processing solutions


• Monitoring and troubleshooting using Datadog, Prometheus, or Azure Monitor


• Experience in DevOps, automation, and infrastructure as code (IaC) for Databricks management

  • Senior Data Engineer

    1 week ago

    Brisbane, Queensland, Australia VALE Partners Full time $120,000 - $180,000 per year

    VALE Partners are working alongside an emerging start-up to find a Senior Data Engineer to join their team. This is a crucial, product-facing role where your work will form the foundation for the real-time analytics and insights customers rely on. The Opportunity: This is a key engineering role where you will be responsible for building and maintaining...

  • Senior Azure Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Talenza Full time $120,000 - $180,000 per year

    Senior Data EngineerTalenza's client, one of the largest employer in Queensland, is looking for a Senior Data Engineer to help them centralising their Data systems. You will be working with well-established and reliable teams, working in a collaborative and supportive environment.Location: Brisbane-based, 3 days / week onsiteEmployment Type: Fixed-term...

  • Data Architect

    1 week ago

    Brisbane, Queensland, Australia Data#3 Full time $120,000 - $180,000 per year

    Data/Information ArchitectDuration:Initial contract to 26 June months) + Potential extensionsLocation:Brisbane, QLD (WFH 2 days)We are seeking an experiencedData/Information Architectto join a large-scale digital transformation program. This contract role will focus on designing and managing enterprise-level data architecture to support the replacement of a...

  • Senior Network Engineer

    1 week ago

    Brisbane, Queensland, Australia Data#3 Limited Full time $120,000 - $180,000 per year

    Join our high performing, and expanding Managed Services TeamBrisbane – Hybrid workingEmployer of Choice – 10 years in a rowData#3 Managed Services team is growing And we are looking for an experienced Senior Network Engineer to join our team.Key ResponsibilitiesProvide customers with value solutions, to meet their needs, within Data#3's delivery...

  • Senior Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Datacom Full time $120,000 - $180,000 per year

    Our PurposeHere at Datacom we connect people and technology in order to solve challenges, create opportunities and discover new possibilities for the communities we live inOur TeamThe Data and Analytics team are a progressive group of diverse and creative individuals with a deep love for all things data related. They build platforms and solutions that are...

  • Senior Data Engineer

    1 week ago

    Brisbane, Queensland, Australia Datacom Full time $120,000 - $180,000 per year

    Our PurposeHere at Datacom we connect people and technology in order to solve challenges, create opportunities and discover new possibilities for the communities we live inOur TeamThe Data and Analytics team are a progressive group of diverse and creative individuals with a deep love for all things data related. They build platforms and solutions that are...

  • Windows Engineer

    5 days ago

    Brisbane, Queensland, Australia NTT DATA Full time $90,000 - $120,000 per year

    Is innovation part of your DNA? Do you want to enable a connected future for people, organizations, and society?Join our growing global NTT team and you'll be part of the world's largest ICT company (by revenue). We've combined the capabilities of 28 remarkable companies to become one, leading technology services provider. Together, we help our people,...

  • Data Engineer

    1 day ago

    Brisbane, Queensland, Australia Humanforce Full time $120,000 - $180,000 per year

    Work options: HybridWe are passionate about people making their own decisions about where and when they work. Our aim is to facilitate hybrid working (mix of in the office and from home) where possible to support our people to be effective, empowered, and productive to achieve both their career and personal goals. Because we recognise that working flexibly...

  • Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Future Fluent Full time $120,000 - $180,000 per year

    The OpportunityFuture Fluent has been exclusively engaged by a leading infrastructure services organisation to appoint aData Engineerin a newly created role in Brisbane.This is a pivotal opportunity to design, build, and maintain scalable, high-performance data infrastructure that underpins the organisation's analytics, machine learning initiatives, and core...

  • Data Engineer

    23 hours ago

    Brisbane, Queensland, Australia Servco Australia Full time $90,000 - $120,000 per year

    Company DescriptionServco Australia is one of the largest automotive retail dealership groups in Australia, operating over 45 locations across QLD, NSW, VIC, and WA. Representing several top automotive brands, Servco employs over 2,000 team members. With core values of respect, service, teamwork, and innovation, we aim to transform the Australian automotive...