Advanced Generative AI Model Architect

1 week ago

Sydney, New South Wales, Australia beBeeResearch Full time $150,000 - $180,000
Job Description

Develop a next-generation evaluation system by leveraging automatic evaluations.

The ideal candidate will engineer sophisticated AI agents that can automatically assess the quality and human alignment of generative design models. This high-impact role focuses on building the practical systems that make cutting-edge research effective, to provide a rapid feedback loop that guides the future of design generation.

This includes:

  • Agentic Evaluation Systems: Engineer autonomous AI agents that use Multimodal Large Language Models (MLLMs) to evaluate the quality, relevance, and human alignment of generated designs.
  • Inference-Time Alignment: Master techniques that improve model outputs without full retraining, but by inference-based methods including prompt engineering, in-context learning and Retrieval-Augmented Generation (RAG).
  • Model Benchmarking & Analysis: Build a rigorous framework to systematically benchmark internal and external quality understanding models, delivering clear, data-driven insights on human alignment.
Primary Responsibilities

The successful candidate will be responsible for:

  • Designing, building, and optimizing the infrastructure for an 'MLLM-as-a-Judge' evaluation system for scalable, automated feedback.
  • Implementing and experimenting with inference-time alignment techniques (Prompt Engineering, RAG, ICL) to directly improve model output quality.
  • Establishing and managing a comprehensive benchmarking process to compare various foundation models on design-centric tasks.
  • Analyzing evaluation data to identify model failure modes and providing actionable recommendations to the research team.
  • Collaborating with research scientists and ML engineers to integrate the agentic judge system into the model development lifecycle.
  • Translating the latest research in LLM evaluation and agentic AI into practical, production-ready engineering solutions.
Required Skills and Qualifications

The ideal candidate should possess:

  • A strong understanding of generative AI models (e.g., Diffusion Models, GANs, Transformers) and their architectures, with practical experience that informs robust evaluation strategies.
  • Excellent skills in creating data-driven evaluation methodologies, turning user analytics into clear, actionable insights.
  • Experience in managing or optimizing large-scale distributed model training across hundreds of GPUs.
  • A solid understanding of machine learning, having worked with PyTorch and knowing how to optimize such codes for speed.
  • Disciplined coding practices, with experience in code reviews and pull requests.
  • Experience working in cloud environments, ideally AWS.
Nice to Have

The following qualifications are desirable:

  • Familiarity with evaluation libraries and frameworks.
  • Experience building or working with agentic AI systems or multi-agent coordination.
  • Knowledge of data visualization tools to communicate findings effectively.
  • A background or interest in human-computer interaction, design principles, or AI ethics.

  • AI Visionary Architect

    2 weeks ago

    Sydney, New South Wales, Australia beBeeArchitecture Full time $150,000 - $250,000

    As a senior architect, you will lead the development of advanced AI solutions across multiple agentic frameworks.Key Responsibilities:Define and drive enterprise AI architecture and strategyGuide adoption of generative AI and multi-agent orchestrationArchitect scalable, secure, and ethical AI/ML solutionsCollaborate with AI/ML engineers to shape use cases...

  • Ai Senior Architect

    1 week ago

    Sydney, New South Wales, Australia Finxl Full time

    AI Senior Architect (Contract) with FinXLFinXL IT Professional Services is a trusted Australian consulting firm, established in 2003, delivering technology-enabled business solutions across Australia and New Zealand.We partner with clients in telecommunications, financial services, government, and beyond to provide expert consulting services and flexible...

  • Ai Senior Architect

    2 weeks ago

    Sydney, New South Wales, Australia Finxl Full time

    AI Senior Architect (Contract) with FinXL FinXL IT Professional Services is a trusted Australian consulting firm, established in 2003, delivering technology-enabled business solutions across Australia and New Zealand.We partner with clients in telecommunications, financial services, government, and beyond to provide expert consulting services and flexible...

  • Advance AI Models in Geography

    2 weeks ago

    Sydney, New South Wales, Australia beBeeGeography Full time $60,000 - $120,000

    Shape the Future of AI in GeographyA unique opportunity awaits geography experts to contribute to the advancements of Artificial Intelligence by generating prompts, defining scoring criteria, and correcting model responses using their domain knowledge.As a geography specialist, you will have the chance to influence future AI models in your field. Your...

  • Ai Senior Architect

    1 week ago

    Sydney, New South Wales, Australia Finxl Full time

    AI Senior Architect (Contract) with FinXLFinXL IT Professional Services is a trusted Australian consulting firm, established in 2003, delivering technology-enabled business solutions across Australia and New Zealand.We partner with clients in telecommunications, financial services, government, and beyond to provide expert consulting services and flexible...

  • AI Senior Architect

    2 weeks ago

    Sydney, New South Wales, Australia FinXL Full time

    AI Senior Architect (Contract) with FinXL FinXL IT Professional Services is a trusted Australian consulting firm, established in 2003, delivering technology-enabled business solutions across Australia and New Zealand. We partner with clients in telecommunications, financial services, government, and beyond to provide expert consulting services and...

  • AI Senior Architect

    2 weeks ago

    Sydney, New South Wales, Australia FinXL Full time

    AI Senior Architect (Contract) with FinXLFinXL IT Professional Services is a trusted Australian consulting firm, established in 2003, delivering technology-enabled business solutions across Australia and New Zealand. We partner with clients in telecommunications, financial services, government, and beyond to provide expert consulting services and flexible...

  • AI Architect

    2 weeks ago

    Sydney, New South Wales, Australia Kaliba Full time US$180,000 - US$190,000 per year

    Ready to Architect the Future of AI?Do you want to design next-generation AI systems that aren't just experiments, but actually power real-world business transformation?Have you mastered the art of turning LLMs, RAG pipelines, and agentic workflows into scalable, secure, high-impact solutions?Are you looking for a place where your architectural vision...

  • Ai Solutions Architect

    7 days ago

    Sydney, New South Wales, Australia Bebeeartificialintelligence Full time

    About Our AI RoleAs we continue to scale and deliver on exciting initiatives, we're making a significant transformation in our Technology function. We pride ourselves on creating the best investing product with UX at the forefront, while breaking barriers in how we elevate services to customers and challenge the status quo.We're seeking an expert AI Software...

  • AI Solutions Architect

    1 week ago

    Sydney, New South Wales, Australia beBeeArtificialIntelligence Full time $150,000 - $200,000

    About Our AI RoleAs we continue to scale and deliver on exciting initiatives, we're making a significant transformation in our Technology function. We pride ourselves on creating the best investing product with UX at the forefront, while breaking barriers in how we elevate services to customers and challenge the status quo.We're seeking an expert AI Software...