Principal Data Architect

We are seeking a Principal Data Engineer to join our team. As a key member of our engineering organization, you will be responsible for designing and developing real-time data pipelines and implementing data mesh architectures to enhance client data interactions.

• Design & Architecture: You will play a key role in designing and architecting scalable solutions using Databricks Lakehouse Architecture, applying best practices around Delta Lake, data modeling, and distributed processing to build robust, future-proof data platforms.
• Developing Data Solutions: Implement and enhance data-driven solutions integrating with clients' systems using state-of-the-art tools such as Databricks, Snowflake, Google Cloud, and AWS. Embrace modern data architecture philosophies including data products, data contracts, and data mesh to ensure a decentralized and consumer-oriented approach to data management.
• Data Pipeline Development: Develop and optimize high-performance, batch and real-time data pipelines employing advanced streaming technologies like Kafka, and Flink. Utilize workflow orchestration tools such as Dataflow and Airflow.
• Database and Storage Optimisation: Optimise and manage a broad array of database technologies, from traditional relational databases (e.g., PostgreSQL, MySQL) to modern NoSQL solutions (e.g., MongoDB, Cassandra). Focus on strategies that enhance data accessibility, integrity, and performance.
• Big Data Processing & Analytics: Utilise big data frameworks such as Apache Spark and Apache Flink to address challenges associated with large-scale data processing and analysis. These technologies are crucial for managing vast datasets and performing complex data transformations and aggregations.
• Cloud Data Management: Implement and oversee cloud-specific data services including AWS Redshift, S3, Google BigQuery, and Google Cloud Storage. Leverage cloud architectures to improve data sharing and interoperability across different business units.
• Security and Compliance: Ensure all data practices comply with security policies and regulations, embedding security by design in the data infrastructure. Incorporate tools and methodologies recommended for data security and compliance, ensuring robust protection and governance of data assets.

To succeed in this role, you will need:

• Databricks Platform: Hands-on experience in Databricks to help us build scalable, high-performance data solutions. You will be responsible for designing and developing end-to-end data pipelines using the Databricks Lakehouse platform, integrating with cloud-native services, and enabling real-time analytics.
• Core Data Engineering Tools & Technologies: Demonstrates proficiency in SQL and Spark, and familiarity with platforms such as Databricks and Snowflake. Well-versed in various storage technologies including AWS S3, Google Cloud BigQuery, Cassandra, MongoDB, Neo4J, and HDFS. Adept in pipeline orchestration tools like AWS Glue, Apache Airflow, and dbt, as well as streaming technologies like Kafka, AWS Kinesis, Google Cloud Pub/Sub, and Azure Event Hubs.
• Data Storage Expertise: Knowledgeable in data warehousing technologies like BigQuery, Snowflake, and Databricks, proficient in managing various data storage formats including Parquet, Delta, ORC, Avro, and JSON to optimise data storage and retrieval.
• Building and Managing Large-scale Data Systems: Experienced in developing and overseeing large-scale data pipelines and data-intensive applications within production environments.
• Data Modelling Expertise: Proficient in data modelling, understanding the implications and trade-offs of various methodologies and approaches.
• Infrastructure Configuration for Data Systems: Competent in setting up data system infrastructures, favouring infrastructure-as-code practices using tools such as Terraform and Pulumi.
• Programming Languages: Proficient in Python and SQL, with additional experience in programming languages like Java, Scala, GoLang, and Rust considered advantageous.
• CI/CD Implementation: Knowledgeable about continuous integration and continuous deployment practices using tools like GitHub Actions and ArgoCD, enhancing software development and quality assurance.
• Testing Tools and Frameworks: Experienced with data quality and testing frameworks such as DBT, Great Expectations, and Soda, ensuring the reliability of complex data systems.
• Commercial Application of Data Engineering Expertise: Demonstrated experience in applying data engineering skills across various industries and organisations in a commercial context.
• Consulting and Advisory Skills: Experienced in a consultancy or professional services setting, offering expert advice and crafting customised solutions that address client needs. Effective in engaging stakeholders and translating business requirements into practical data engineering strategies.

This is an excellent opportunity to work with a talented team of engineers and contribute to the success of our organisation. We offer a competitive salary and benefits package, as well as opportunities for career growth and professional development.

Professional Experience and Qualifications: At least 8+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment. Consulting experience within a technology consultancy or professional services firm is highly beneficial.