Lead Data Engineer

Overview
Endava Perth, Western Australia, Australia is seeking aLead Data Engineerspecialising in Databricks to design, build, and optimise data pipeline solutions on Databricks and related cloud platforms.
You will work closely with data scientists, analysts, and engineers to ensure our data infrastructure supports advanced analytics and business insights across industries (including energy, resources, and mining).
You will join a collaborative, agile team where continuous improvement, innovation, and knowledge sharing are part of the culture.
Responsibilities
Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective pipeline solutions.
Contribute to Data Architecture and Solution Design, helping to build proofs of concepts.
Design, develop, and maintain robust ETL/ELT pipelines on Databricks, using AWS, Azure, and GCP tools and services to ingest, process, and transform large datasets.
Implement data validation, cleansing, and governance procedures to guarantee data quality, integrity, and security, including enforcing data standards and proactively addressing data quality issues.
Continuously improve the scalability, efficiency, and cost-effectiveness of data pipelines and identify opportunities to enhance performance, reliability, and cost-efficiency across data systems.
Monitor data pipeline performance and troubleshoot issues to ensure high data availability and consistency; leverage observability tools and best practices to maintain reliable pipelines.
Develop streaming or event-driven data processes for real-time analytics, leveraging frameworks like Apache Kafka and Spark Structured Streaming.
Maintain clear documentation of data pipelines, data models, and processes, and follow best practices in coding, testing, and version control to ensure maintainable and auditable workflows.
Qualifications
Proficiency in Python for data engineering (including PySpark and libraries like pandas/Polars) and SQL for data querying and transformation.
Solid understanding of data warehousing concepts and dimensional data modeling (e.g. star schema, Kimball methodology).
Hands-on experience with relational database systems and SQL (e.g. SQL Server, PostgreSQL) and familiarity with NoSQL databases (e.g. MongoDB, Cassandra).
Strong experience designing and implementing ETL/ELT processes and integrating data from multiple sources.
Proven experience working with multiple cloud data platforms such as AWS, Azure, and GCP.
Expertise in Databricks and the Spark ecosystem for large-scale data processing.
Familiarity with data pipeline orchestration and automation tools and with CI/CD pipelines for deploying data workflows.
Experience monitoring data pipeline performance and using observability tools to ensure data reliability is a plus.
Experience designing and building event-driven architectures and streaming data tools (such as Apache Kafka or Spark Streaming) is beneficial for real-time data flows.
Experience working in Agile teams with iterative development and a collaborative problem-solving approach.
Holding a current Databricks certification (e.g., Databricks Certified Data Engineer) is a strong advantage.
Background in or understanding of data from the energy, resources, or mining industry is a plus.
Benefits
Finance: Competitive salary package, share plan, company performance bonuses, value-based recognition awards, referral bonus
Career Development: Career coaching, global career opportunities, non-linear career paths, internal development programmes for management and technical leadership
Learning Opportunities: Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platform subscriptions
Work-Life Balance: Hybrid work and flexible working hours, employee assistance program
Health: Global internal wellbeing programme, access to wellbeing apps
Community: Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, events and celebrations
Additional Local Benefits
Monthly Lifestyle Allowance: Contribution towards health and wellbeing activities like gym memberships
Novated Leasing: Pre-tax car leasing benefit for new and used cars
Loyalty Leave: Additional leave on 3rd, 4th, and 5th work anniversaries, up to 3 extra days per year
Inclusive Parental Leave Policy: 12 weeks primary carer's leave and 4 weeks secondary carer's leave
Work From Anywhere: 20 days of working from anywhere per year in addition to the hybrid policy
At Endava, we're committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best.
We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers.
Hiring decisions are based on merit, skills, qualifications, and potential.
If you need adjustments or support during the recruitment process, please let us know.
Seniority level
Mid-Senior level
Employment type
Full-time
Job function
Information Technology
Industries
IT Services and IT Consulting
#J-18808-Ljbffr