Senior Data Engineer

6 days ago

Brisbane, Queensland, Australia Celebal Technologies Full time $120,000 - $200,000 per year

Celebal is looking for Data Engineer/Senior Data Engineer.

Company Profile
- Celebal Technologies a Premier Microsoft Azure partner and have been recently funded by Norwest Venture Capital. Celebal Technologies is a premier software services company in the field of Data Science, Big Data, Enterprise Cloud & Automation. Established in 2016, in this short span of time we grew to headcount of 2200+ We help in achieving a competitive advantage with intelligent data solutions, built using cutting-edge technology.

Key Responsibilities:


• Lead the implementation of data ingestion pipelines, ensuring optimized data flow and transformation logic.


• Oversee and guide the migration of DBT ETL scripts, rewriting or optimizing SQL and PySpark code for Databricks.


• Expertise in DBT, SQL Server, Synapse (T-SQL, performance tuning) and data migration to cloud platforms.


• In-depth knowledge of Databricks, Spark, Delta Lake, and integration with Azure services.


• Design and implement reusable frameworks for data transformations, logging, and error handling in Databricks.


• Strong understanding of cloud-based architecture for data lakes, data warehouses, and scalability.


• Expertise in Azure Data Factory (ADF), Synapse Analytics, and Azure Blob Storage integration with Databricks.


• Experience in implementing and managing Medallion architecture (Bronze, Silver, Gold) in Databricks.


• Optimize Spark configurations, cluster sizing, and execution strategies for performance improvements.


• Conduct code reviews to ensure adherence to best practices, performance optimizations, and security guidelines.


• Debug and troubleshoot issues related to data ingestion, transformation, and job failures in Databricks.


• Collaborate with data engineers and analysts to ensure transformed data meets business requirements and reporting needs.


• Implement robust logging and monitoring using tools such as Datadog, Prometheus, or Azure Monitor.


• Work closely with cloud and DevOps teams to enable infrastructure as code (IaC) for Databricks workspace management.


• Provide guidance on version control strategies, branching, and release management for data pipeline


• Act as a bridge between the Solution Architect and the development team, ensuring seamless implementation.


• Ensure data quality by implementing validation checks, data reconciliation, and automated data profiling.


• Strong collaboration and communication skills for leading cross-functional teams and reporting to stakeholders.


• Document all technical aspects, including ETL flows, DevOps workflows, and migration steps for knowledge sharing.


• Mentor junior engineers, conduct knowledge-sharing sessions, and enforce coding standards across the team.

Technical Skills:


• Expertise in Databricks, Spark, PySpark, Delta Lake, and SQL


• Strong hands-on experience with DBT, SQL Server, and T-SQL performance tuning


• Proficiency in Azure cloud services (Data Factory, Synapse, Blob Storage, etc.)


• CI/CD implementation for data pipelines using Azure DevOps, Terraform, or GitHub Actions


• Experience with Medallion architecture and scalable data processing solutions


• Monitoring and troubleshooting using Datadog, Prometheus, or Azure Monitor


• Experience in DevOps, automation, and infrastructure as code (IaC) for Databricks management

  • Senior Data Engineer

    6 hours ago

    Brisbane, Queensland, Australia meshynix Full time $120,000 - $180,000 per year

    Position:Senior Data EngineerLocation:Brisbane (On-Site)Duration:12-month initial contractWorking Flexibility:HybridCompany DescriptionMeshynix is a purpose-led firm of Databricks experts committed to delivering meticulously designed and highly scalable Data Lakehouse solutions. Operating out of Australia and India, Meshynix helps organizations worldwide to...

  • Senior Data Engineer

    4 days ago

    Brisbane, Queensland, Australia Latitude IT Full time $104,000 - $155,000 per year

    Senior Data EngineerLocation:Brisbane City - HybridIn-Office:1-2 days per week with flex on daysPackage:Up to $155k base + super + 10% bonusThe Opportunity:One of Australia's leading QSR brands seeks a talented Senior Data Engineer to take ownership of a Snowflake and Azure Data Factory landscape that sits behind millions of customer orders.This is a...

  • Senior Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia VALE Partners Full time $120,000 - $180,000 per year

    VALE Partners are working alongside an emerging start-up to find a Senior Data Engineer to join their team. This is a crucial, product-facing role where your work will form the foundation for the real-time analytics and insights customers rely on. The Opportunity: This is a key engineering role where you will be responsible for building and maintaining...

  • Senior Data Engineer

    3 hours ago

    Brisbane, Queensland, Australia Power Tech Consulting Full time $120,000 - $180,000 per year

    Senior Data EngineerLocation: Onsite @Brisbane location (with hybrid working arrangements)Type: Full-Time | PermanentExperience Level: 5+ yearsAustralian Working Rights: Fulltime working rights are required with Valid VisaAbout the RoleWe're seeking a seasoned Data Engineer to build and deliver the data platforms that empowers intelligent decision-making....

  • Data Architect

    2 weeks ago

    Brisbane, Queensland, Australia Data#3 Full time $120,000 - $180,000 per year

    Data/Information ArchitectDuration:Initial contract to 26 June months) + Potential extensionsLocation:Brisbane, QLD (WFH 2 days)We are seeking an experiencedData/Information Architectto join a large-scale digital transformation program. This contract role will focus on designing and managing enterprise-level data architecture to support the replacement of a...

  • Senior Network Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Data#3 Limited Full time $120,000 - $180,000 per year

    Join our high performing, and expanding Managed Services TeamBrisbane – Hybrid workingEmployer of Choice – 10 years in a rowData#3 Managed Services team is growing And we are looking for an experienced Senior Network Engineer to join our team.Key ResponsibilitiesProvide customers with value solutions, to meet their needs, within Data#3's delivery...

  • Senior Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia Datacom Full time $120,000 - $180,000 per year

    Our PurposeHere at Datacom we connect people and technology in order to solve challenges, create opportunities and discover new possibilities for the communities we live inOur TeamThe Data and Analytics team are a progressive group of diverse and creative individuals with a deep love for all things data related. They build platforms and solutions that are...

  • Windows Engineer

    1 week ago

    Brisbane, Queensland, Australia NTT DATA Full time $90,000 - $120,000 per year

    Is innovation part of your DNA? Do you want to enable a connected future for people, organizations, and society?Join our growing global NTT team and you'll be part of the world's largest ICT company (by revenue). We've combined the capabilities of 28 remarkable companies to become one, leading technology services provider. Together, we help our people,...

  • Data Engineer

    4 days ago

    Brisbane, Queensland, Australia Humanforce Full time $120,000 - $180,000 per year

    Work options: HybridWe are passionate about people making their own decisions about where and when they work. Our aim is to facilitate hybrid working (mix of in the office and from home) where possible to support our people to be effective, empowered, and productive to achieve both their career and personal goals. Because we recognise that working flexibly...

  • Principal Data Engineer

    2 weeks ago

    Brisbane, Queensland, Australia SS&C Technologies Full time $120,000 - $180,000 per year

    As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000+ employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale, and technology.*Job...