Package index • fathom

Building an instrument

An instrument bundles the questions with the labels the model may answer with. Read the assembled prompt before spending anything on a run.

instrument(): Construct a validated instrument of items
template_items(): Build items by substituting categories into one question stem
preview(): Print the assembled prompt
ollama_backend(): Configure an Ollama backend

Extracting

One unconstrained call per document-item, reading the probability distribution at generated position 1.

measure(): Measure documents against an instrument
read_checkpoints(): Read a checkpoint directory back into results

Reading the results

Aggregate statistics cannot tell you whether an item measures what you intended. These describe; only inspect_cases() answers that question, by showing you the documents.

item_report(): Summarize how each item behaved
inspect_cases(): Read the documents that fired on an item

Preparing a corpus

A corpus of files rather than a column of text. One row per document, with the columns measure() and the audit functions expect.

prepare_image(): Turn a folder of images into a corpus measure() can take
prepare_audio(): Turn recordings into a corpus of annotated transcripts
prepare_video(): Turn videos into a corpus carrying both what was shown and what was said

Testing the measurement

A probability is more than a label, and these are what that buys. How much of an item’s endorsement rate is the threshold rather than the corpus, whether the model agreed with a human, how far two humans agreed with each other, and whether its probabilities were honest.

threshold_sensitivity(): Compute endorsement rates across a range of thresholds
coding_sample(): Draw a blinded sample for human coding
validate_sample(): Validate a completed coding sheet against the model
coder_reference(): Collapse several coders into one reference sheet

Downstream

as_wide(): Pivot results into an analysis matrix

Data

Two corpora hand-coded by the researchers who collected them. parlasent is the only one carrying two independent codings before reconciliation, so it is the only one against which a model can be read at a human ceiling rather than against perfect agreement. It ships under CC BY-SA 4.0 rather than MIT, see LICENSE.note.

ati: Mexican access-to-information requests
parlasent: UK parliamentary sentences, coded for sentiment on a six-level scale

Package

fathom fathom-package: fathom: Qualitative Coding and Measurement with Local Language Models