Galaris

The Lab: compare on your work cases

Compare models on your own work cases. Prepare experiments with an agent, inspect evidence and assess results against your own judgment.

Considering a different model or harness? The Lab helps you measure differences on your own work cases. Compare models and mechanism configurations in comparable contexts, then check the selected engine in use. These evaluations do not replace a full trial of tools and external effects. Choose models and harnesses.

Ten mechanisms to examine separately

Family Mechanisms
Prepare work Dispatcher, Planner
Organise and learn Topic detection, memory extraction, learning
Follow progress Goal follow-up
Execute Task, conversation and voice executors
Diagnose Task analysis

The incident journal is separate.

The AI-mechanism laboratory — The ten mechanisms available in the Lab and their uses.
The AI-mechanism laboratory — The ten mechanisms available in the Lab and their uses.C43

Build a dataset around a question

To start a dataset, generate 1 to 20 synthetic cases suited to the mechanism and, optionally, the experiment context. They are saved as drafts for review, with expected outcomes and generation provenance. An incomplete generation does not publish a partial dataset.

Create mechanism-specific cases with a tested variable, context and expected result. Capture a Task, round or voice turn with provenance. Differences between source configuration and dataset parameters are visible before import.

Separate work, validation and holdout sets. Categorise nominal cases, ambiguity, missing context, multilingual input, robustness, real incidents or valid alternatives. Incomplete cases remain drafts. Model-proposed references need review.

Execute, then judge

Select candidate and judge separately. Launching fixes cases, parameters and models. The first pass produces outputs and objective checks; the second judges them. The candidate never receives the expected result.

Inspect scores by dimension, explanations, critical failures and unjudged cases. A finished campaign is not necessarily successful; a judge failure leaves a missing score.

Measure stability and resume work

Repeat each item from 1 to 20 times to compare success rate, average, extremes and spread. Cancel while preserving published results, resume remaining items with the snapshot, or rejudge saved outputs without calling the candidate.

An optional budget covers candidate and judges when admitting new steps. Repetition measures stability on these cases, not every future situation.

Compare before and after a change

All ten mechanisms support comparing two evaluations, including evaluations from different datasets. Galaris pairs identical inputs, references and repetitions to display outputs, criteria, costs and durations side by side, or stacked on mobile. The summary covers complete runs. Ambiguous pairing or incompatible configurations block misleading improvement conclusions.

Regressions to review highlights new critical failures, cases moving from pass to fail and changes on shared criteria. Latency, cost and quality shows medians and available pair counts, including the first observed output outside queue and judge time. Missing measurements and scores remain missing.

Comparison uses recorded results without running inference or changing production settings. Refresh it after new results or judgments. It helps examine a change on your cases; it does not establish statistical superiority on its own.

The summary distinguishes unique cases from repetitions, comparable pairs, missing outputs, unjudged results and errors. Repeating a case does not create independent cases; a missing output does not become a zero score.

Add independent human review

Review can hide model identity and automated scores. Each reviewer scores and explains their decision before revealing the judgment. Disagreements become visible. A revealed review is fixed; rejudging opens another campaign.

A test and its Lab assessment — Two synthetic cases actually run, with results and recorded costs.
A test and its Lab assessment — Two synthetic cases actually run, with results and recorded costs.C52

Prepare and analyse experiments with an agent

Enable the Galaris Lab connection so an agent can prepare datasets, run evaluations, compare results and inspect evidence through the laboratory’s 50 MCP functions. It works on the same experiments you see in the interface. Long analyses have inspectable progress.

The connection is disabled by default. The Galaris Lab skill can guide an agent but does not replace permissions; capturing and diagnosing real work also requires administrative inspection access. Editing an experiment does not change production prompts or models.

An agent review retains its author identity and remains distinct from human review and the automated judge. Ask for help exploring results while keeping your own evaluation.

Turn an incident into a check

Analyse a Task dossier with human context without replaying execution. Capture the diagnosis to evaluate that mechanism too.

Simulated tools do not prove external effects. The voice Lab evaluates a transcript, not STT/TTS acoustic quality. Incident tracking complements this evaluation.

Evaluate genuinely permitted choices

The Dispatcher Lab examines the modes allowed by the harness and retains deterministic decisions and their reasons. The Lab’s ten evaluable mechanisms are separate from Dream’s fourteen background mechanisms.