AI-scored hiring and assessment tools. Eight criteria.
Published 2026-09-22. Fixed before any vendor entry was scored.
Scope
This rubric assesses what a vendor publishes about the validity and fairness of its scores.
It does not assess the tool, the model, the underlying science or the company. A vendor whose instrument is strong and whose public documentation is limited will show few criteria as documented, and no assessment of that instrument is made or implied. The reverse also holds.
Reference standards
Each criterion is derived from a section of a bias and measurement-equivalence audit. The sections are identification, sample, grouping variables tested, invariance testing, items flagged for differential item functioning and their disposition, partial invariance, adverse impact, reliability by group, disposition and sign-off, limitations, and next audit.
Each criterion also maps to a technical-documentation item of Annex IV of the EU AI Act, and rests on a chapter of the Standards for Educational and Psychological Testing (AERA, APA and NCME). Chapters are cited by title.
Levels
| Level | Definition |
|---|---|
| Documented | The information is publicly available, specific and sourced, and is reachable without a sales conversation. |
| Partial | Information is published but is general, undated, unquantified, or reachable only on providing contact details. |
| Absent | No public source was found by the search protocol below on the stated access date. |
Absent records the outcome of a defined search on a defined date. Where a source is supplied that the search did not reach, the cell is re-scored under the corrections policy, with the date of the change recorded and the previous level retained.
Each vendor entry reports the number of criteria at each level. No weighted total is calculated and entries are listed alphabetically.
Search protocol
Applied identically to each vendor.
- The vendor domain: science, research, validity, technical, trust, legal, compliance and resources sections, and the site search.
- Documents linked from those pages, including documents that require contact details. Where a document requires contact details it is recorded as existing, as requiring contact details, and as not read.
- Published bias-audit material, including any New York City Local Law 144 summary, which that law requires to be posted publicly.
- One general web search per vendor, on the vendor name with each of: technical manual, validity, adverse impact, bias audit, reliability.
- Sources behind a login, information obtained in a sales conversation or through personal contact, and descriptions by third parties are excluded.
Each cell records a verbatim quotation, the source URL and the access date, or records the level as Absent. No cell is completed by inference.
C1. Is a technical manual or validity report publicly available without a sales conversation?
Audit standard: publication rule; section 1, identification. Annex IV: item 1, general description, intended purpose and instructions for use. Standards: the chapter on supporting documentation for tests.
| Level | Definition |
|---|---|
| Documented | A technical manual, validity report or equivalent is downloadable from the public site without a form or a login, and is dated. |
| Partial | Such a document exists but requires contact details, is undated, or is a summary of a document that is not itself published. |
| Absent | No such document is offered publicly. Blog posts and product pages are not technical documentation. |
Example. A dated document headed Technical Manual, downloadable on click, is Documented. The same document behind a contact form is Partial, and the cell records that it exists and was not read. A page describing the science behind an assessment that carries no data is Absent for this criterion.
C2. Are validity coefficients reported with sample sizes and uncertainty?
Audit standard: section 2, sample. Annex IV: item 2(g), validation and testing procedures and accuracy metrics; item 4, appropriateness of the performance metrics. Standards: the chapter on validity.
| Level | Definition |
|---|---|
| Documented | At least one criterion-related validity coefficient is published with its sample size, its criterion, and an interval or standard error. |
| Partial | A coefficient is published without a sample size, without a named criterion, or without an uncertainty measure; or validity is described in words only. |
| Absent | No coefficient is published publicly. |
Example. A coefficient reported with its sample size, its criterion and a confidence interval is Documented. A comparative marketing claim carrying no coefficient, sample size or criterion is Absent. A coefficient reported alone is Partial.
C3. Is adverse impact reported by subgroup, with the subgroups named?
Audit standard: section 7, adverse impact; section 3, grouping variables tested. Annex IV: item 3, foreseeable unintended outcomes and sources of risk. Also New York City Local Law 144, which requires the summary of an independent bias audit to be posted publicly. Standards: the chapter on fairness in testing.
| Level | Definition |
|---|---|
| Documented | Impact ratios or selection rates are published for named subgroups, with the groups, the reference group and the period stated. |
| Partial | Fairness is described, or a bias audit is stated to have been conducted, without published subgroup figures; or the groups are not named. |
| Absent | No subgroup analysis is published. |
Example. A posted bias-audit summary giving dated impact ratios by sex and by race or ethnicity is Documented. A statement of independent auditing with no figures and no named auditor is Partial.
A vendor without United States exposure is not subject to Local Law 144. Absent on this criterion is a record of what is published and is not a finding of non-compliance with any obligation.
C4. Is the scored construct named and defined, rather than described only by outcome?
Audit standard: section 1, identification. Annex IV: item 1(a), intended purpose; item 2(b), design specifications and the general logic of the system. Standards: the chapter on validity, on evidence based on test content and on internal structure.
| Level | Definition |
|---|---|
| Documented | The tool states what it measures in construct terms, defines each construct, and states how the construct maps to the score reported. |
| Partial | Traits are named but not defined, or the naming is proprietary with no published definition or referent. |
| Absent | The tool is described only by what it predicts or by what it provides to the buyer. |
Example. A named construct, defined by reference to a published model and reported against a stated norm, is Documented. A description of the tool in terms of the candidates it surfaces, naming no construct, is Absent. Proprietary trait names without definitions are Partial.
The finding under this criterion is that a construct is not publicly defined. It is not a finding that no construct exists or that the score measures nothing.
C5. Is there evidence of comparability across languages, versions or delivery modes?
Audit standard: section 4, invariance testing; section 5, items flagged for differential item functioning and their disposition; section 6, partial invariance. Annex IV: item 2(d), training data characteristics and provenance; item 6, changes through the lifecycle. Standards: the chapter on fairness in testing, and its standards on test adaptation and translation.
| Level | Definition |
|---|---|
| Documented | Measurement equivalence, differential item functioning or an equating study is published across languages, versions or modes, with the groups, the method and the outcome stated. |
| Partial | Multiple languages or versions are offered and comparability is described, or translation quality is described, without equivalence evidence. |
| Absent | Multiple languages or versions are offered and nothing is published on comparability. |
Example. A published invariance cascade across language versions, with released items and their disposition, is Documented. A statement that an instrument is available in many languages and professionally translated is Partial: translation quality is a process description rather than evidence that scores are comparable.
C6. Is human oversight documented, with a candidate route to contest a score?
Audit standard: section 9, disposition and sign-off. Annex IV: item 2(e), human oversight measures; item 3, the technical measures for human oversight. Also Local Law 144, on candidate notice at least ten business days before use and on a published instruction for requesting an alternative selection process. Standards: the chapter on the rights and responsibilities of test takers.
| Level | Definition |
|---|---|
| Documented | A candidate-facing statement records that a score is produced, states who reviews it, and gives a named route to query, contest or request an alternative. |
| Partial | Human involvement is described to buyers, or Article 22 of the GDPR is cited, without a candidate-facing route to contest the score. |
| Absent | Nothing candidate-facing is published on oversight or on contesting a score. |
Example. A candidate page recording that responses are scored automatically, naming the reviewing role and giving an address for queries, is Documented. A privacy notice stating that no solely automated decision-making takes place addresses a different question and is Partial.
C7. Is any independent review named, and is the reviewer identified?
Audit standard: independence rule, under which analysis by the instrument owner meets professional practice but does not meet the Local Law 144 independence test, and external review is required before sale into a jurisdiction with an audit rule. Annex IV: item 7, standards applied; item 8, the EU declaration of conformity. Standards: the chapter on supporting documentation for tests.
| Level | Definition |
|---|---|
| Documented | An external reviewer or auditor is named, with the date and scope of the review, and the vendor states the reviewer’s independence. |
| Partial | An audit, certification or review is described with no named reviewer, no date or no scope; or the reviewer is a related party. |
| Absent | No external review is described. |
Example. A named auditor, with the date and scope of the audit and a posted summary, is Documented. A statement of independent validation with no name is Partial. An internal study is Absent for this criterion, which records only whether an external review is published.
C8. Is reliability reported, and reported by group?
Audit standard: section 8, reliability by group. Annex IV: item 2(g), performance metrics; item 3, degrees of accuracy. Standards: the chapter on reliability and precision, and errors of measurement.
| Level | Definition |
|---|---|
| Documented | Reliability is published per scale with its sample size and the coefficient named, and is broken out by at least one subgroup or language. |
| Partial | Overall reliability is published without a subgroup breakout, or a coefficient is given without a sample size or without attribution to a scale. |
| Absent | No reliability evidence is published. |
Example. A table of internal-consistency and test-retest coefficients per scale, with sample sizes, split by language version, is Documented. A coefficient given for an unnamed set of scales is Partial.
Outside the scope of this rubric
- The quality of any tool.
- Compliance with the EU AI Act. The Annex III obligations for stand-alone high-risk systems apply from 2 December 2027, under the Digital Omnibus adopted on 29 June 2026. Annex IV is used here as a documentation checklist. No vendor is described as non-compliant with an obligation that is not in force.
- Pricing, marketing and corporate conduct.
Corrections
Write to milos@centerforpsychology.me. Every correction is answered within ten working days. Anyone may write, including the vendors assessed.
A correction that changes a level is applied to the entry and to the published data, and the entry states what changed and when. The previous level is retained. A correction that is not accepted receives a reasoned answer. Nothing is quietly amended.