Skip to content
LabsTrail

How the reproducibility score works

Each rating asks whether a paper reports what others would need to repeat its work. The score runs from 0 to 100, and it doesn't say whether the findings are right.

LabsTrail takes papers in biology, medicine and health. A paper outside the field is refused before it is published, and a rating asked for one is cancelled.

How a paper is scored

  1. 1

    The model reads the paper

    The rating fetches the paper's PDF from IPFS by its CID, the one the submitter signed. gemini-3.8-flash reads its text and decides which kinds of study it reports. One paper can combine several, such as experiments on mice and lab work on cells.

  2. 2

    It answers each check with a quote

    For each check those kinds of study call for, it answers shown, partly or missing, and quotes the passage it relied on.

  3. 3

    Code counts the points

    A check the paper shows earns a point, one it partly shows half a point, a missing one none, and an answer counts only when its quote is found in the paper's text. The score is the points over the checks that apply, out of 100.

A checklist for each kind of study

The checks are written in our own words from CONSORT 2025, STROBE, ARRIVE 2.0, MDAR, PRISMA 2020, SANRA and the DOME recommendations. How strictly each check applies to a kind of study follows the Rigor and Transparency Index (Menke and colleagues, iScience 2020). A paper that reports several kinds of primary research, such as an animal study with lab work, answers the checks of each, every check once.

  • A randomized controlled trial in peopleCONSORT 202515 checks
  • An observational study of people: cohort, case-control or cross-sectionalSTROBE14 checks
  • An experiment on live vertebrates, such as mice, rats or zebrafish, or on cephalopodsARRIVE 2.013 checks
  • A laboratory study of cells, tissues, molecules or microbes, or of invertebrates such as flies or wormsMDAR14 checks
  • A systematic review or meta-analysis of published studiesPRISMA 202013 checks
  • A narrative review of a topic, without a systematic searchSANRA12 checks
  • A computational study: bioinformatics, modelling or machine learning on dataDOME12 checks
  • Any other kind, such as a case report, a qualitative study, a protocol or a diagnostic accuracy studyGeneral checks8 checks

What the score leaves out

  • A made-up paper that states every right thing would score well.
  • The model can misread a paper. Each answer and its quote are on the rating's page, so you can check them.
  • A very long paper is shortened to fit the model, and its rating's page says so.

Where it is recorded

The score goes on Base, in LabsTrail's RatingController contract. The answers and quotes go in the rating's own Knowledge Asset on the OriginTrail DKG, which points to the paper's. Like the paper's, the rating's statements are anchored on Base by their merkle root, so anyone can read them and check they haven't changed since. The paper's asset never changes: a new rating adds its own score, and the paper's page shows the latest.

Three phases, three scores

Each phase records its own score next to the earlier ones, which stay as they were.

  1. Reproducibility readiness

    live

    The score on this page, from an AI model against the checklists for the kinds of study the paper reports.

  2. Human review

    next

    Reviewers drawn at random with Chainlink VRF, so anyone can check the draw, read the paper and score it. Their scores join the rating's record, in the open.

  3. Wet-lab replication

    later

    A lab repeats the experiments, then reviewers check its results.

Pick a paper to rate