Evaluation and Quality Engineering
Human Evaluation — Stack
Human evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
What is being assembled is a review workflow: a queue, a task interface, a guideline, an agreement computation, and a path from a verdict back into the evaluation record.
- Task-type fit. Whether the tool natively supports the comparison or annotation form you need — side-by-side preference, span highlighting, multi-criterion forms — rather than only a rating widget.
- Multi-reviewer and adjudication support. Whether an item can be routed to several reviewers and disagreements escalated, without that being a manual process.
- Agreement statistics. Whether inter-rater agreement is computed by the tool and visible continuously, rather than exported and calculated once.
- Guideline versioning. Whether the instructions shown to reviewers are versioned and recorded on each verdict, since a guideline edit invalidates comparability.
- Queue composition control. Whether items can be routed by stratum, difficulty or flag, so expensive reviewers are not spent on easy cases.
- Reviewer management. Whether throughput, agreement with consensus, and attention checks are tracked per reviewer, which is how a quietly degrading reviewer is noticed.
- Blinding. Whether the interface can conceal model identity and randomise candidate order by default rather than by convention.
- Data handling. Whether production content can be shown to external reviewers at all under your obligations, and what redaction the tool supports before it is displayed.
- Reviewer welfare controls. For harmful-content review, whether exposure can be limited, content blurred by default, and session length capped.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Dedicated annotation platforms (Label Studio, Argilla, Prodigy, Doccano and comparable) | Structured annotation with multiple reviewers, adjudication and agreement built in | Oriented around labelling campaigns; the long-lived review programme and its reporting are yours to assemble |
| Human-review features of LLM evaluation platforms (LangSmith, Langfuse, Braintrust and similar) | Reviewing exactly the traces the automated run produced, in the same place the scores live | Review interfaces are usually simpler than dedicated tools; limited adjudication and agreement machinery |
| Managed annotation workforces and data-labelling vendors (Scale, Surge, Toloka, Appen and comparable) | Volume, surge capacity and access to recruited specialists you cannot hire | Quality varies with how well the guideline is written; the vendor cannot know your domain; data residency is a contract question |
| General crowdsourcing marketplaces (Amazon Mechanical Turk, Prolific and similar) | Preference judgements on general-purpose output where no expertise is required | No domain competence; task design and attention checks carry the entire quality burden |
| An internal review tool over your own data store | Organisations where production content may not leave the environment, or where reviewers are existing staff | Someone owns it forever, and reviewer-facing polish is always the lowest priority on that backlog |
| Spreadsheet-based review | A single small campaign to find out whether the questions are the right ones | No blinding, no randomisation, no agreement computation, and no way to stop reviewers seeing each other's answers |
| Support and quality-assurance tooling already used by operations teams | Programmes where the reviewers are support staff already reviewing conversations for other reasons | Scoring rubrics are oriented to agent performance management, not to model quality |