Thesis linters¶

Disclaimer — read before running. These scripts were written with substantial help from large language models (AI coding assistants) and are provided as-is, with no warranty of any kind — you run them at your own risk. They may contain bugs or behave unexpectedly, and their findings are suggestions, not authoritative. Before running them:

Verify the files are the ones published here — check them against the shipped SHA256SUMS (shasum -a 256 -c SHA256SUMS).

Scan them for insecure code — run Bandit, the standard Python security linter (pip install bandit && bandit -r .), and/or the quick grep pattern check.

Review the code yourself — it is short and readable by design, and keep backups of anything you point the linters at.

Full details in Verify what you downloaded before running it. The maintainers accept no liability for any loss or damage arising from use of these scripts.

New here? Start with this¶

A linter is a small program that reads your writing and points out likely problems — the way a spell-checker underlines misspellings, but for the things that matter in a thesis: undefined acronyms, figures nobody refers to, vague claims, references that don't check out, and so on. This folder holds about thirty such linters, one per kind of problem. You run them on your compiled thesis PDF (or your LaTeX source), and each one prints a list of things to look at.

Two things to know before you start:

The findings are suggestions, not a grade. A linter flags what might be wrong so you can decide. Some flags are false alarms — that's expected. Nothing here is submitted or seen by anyone but you unless you share it.
You don't need to understand every linter to benefit. Run the whole set with one command (below), skim the report, fix what's clearly right, and ignore the rest. Come back to the detailed reference when you want to know exactly what a specific check does.

What you need: your thesis as a PDF, Python 3 installed, and a terminal. The Quick start gets you a first report in about two minutes. The rest of the page is reference material you can read as questions come up.

These linters check MSc thesis manuscripts against the writing instructions of ml-theses.org — the thesis guide for students supervised by Alex Jung at Aalto University. (Writing a conference or journal paper instead of a thesis? Add --profile paper — see Profiles.)

Every script is run as python3 <script> ... and prints a findings report. The exit status is 0 when the manuscript is clean, 1 when there are findings, and 2 on a usage error (bad arguments, missing file), so you can also wire the linters into scripts or CI.

Prefer a visual report? Add --dashboard to the runner — python3 run_all_linters.py thesis.pdf --dashboard opens a self-contained HTML dashboard in your browser (summary band, one card per linter, the figure × ten-rules matrix). The dashboard shows only the linters you ran: without --llm it contains just the fast heuristic checks — the semantic assessments (research questions, contributions, story flow, the thesis checklist) appear only when you add --llm. See Basic usage, or view a live example dashboard (a real --llm --bib run of this suite on a sample document).

Quick start¶

# 1. get the suite
git clone https://github.com/alexjungaalto/masterthesis.git
cd masterthesis/assets/linters

# 2. one-time setup
python3 -m venv .venv && source .venv/bin/activate
pip install pymupdf                       # some PDF linters need it

# 3. fast heuristic pass (no LLM, no network) over a compiled PDF
python3 run_all_linters.py thesis.pdf         # replace thesis.pdf with your file

# 4. recommended: add the LLM + bibliography linters — the semantic checks
#    (research questions, contributions, flow, checklist) run ONLY with --llm
export AALTO_API_KEY=...                   # key from the Aalto API dev portal
python3 run_all_linters.py thesis.pdf --llm --bib

# 5. optional: get an HTML dashboard, opened in your browser
#    (shows only the linters you ran — combine with --llm for the full picture)
python3 run_all_linters.py thesis.pdf --llm --bib --dashboard

A few terms in those commands, in plain language:

Compiled PDF — the finished PDF you get when you build your LaTeX project (or export from Word/Overleaf). Point the linters at that file, named however you like; thesis.pdf above is just a placeholder.
Fast / heuristic pass (step 3) — the linters that use only simple text rules. No internet, no API key, nothing leaves your computer. Start here.
LLM linters (step 4) — the checks powered by a large language model (the same kind of AI as ChatGPT). This is where the substantive feedback lives: whether your research questions are well-posed and answered, whether the claimed contributions are supported by the text, story flow, and the thesis checklist. None of these run without --llm — the fast pass in step 3 covers only mechanical issues. The trade-off: --llm needs an API key and sends your text to a language-model service; see Data handling for where that text goes and how to keep it private. --bib turns on the reference checker, which looks each citation up online.
Dashboard (step 5) — the same results as a web page instead of terminal text (see Basic usage). It renders whatever linters you ran, so pair it with --llm to see the semantic assessments as cards too.

Once you have a report, Basic usage explains how to read a finding line (severity, code, location, evidence). Everything below that is reference — dip into it as questions come up.

Getting the scripts¶

All linters live in one directory — assets/linters/ in the masterthesis repo. Two ways to get them:

The whole suite (recommended) — clone or download the repo and work inside assets/linters/:

git clone https://github.com/alexjungaalto/masterthesis.git
cd masterthesis/assets/linters

A single script — each one is served on the website at its own URL, e.g. https://ml-theses.org/assets/linters/prose_lint.py. Download it (and the shared helper) with curl:

curl -O https://ml-theses.org/assets/linters/prose_lint.py
curl -O https://ml-theses.org/assets/linters/lintutil.py   # shared helper
curl -O https://ml-theses.org/assets/linters/aalto_llm.py  # for *_llm scripts

Most linters import lintutil.py, and the *_llm ones import aalto_llm.py (a few linters use only one of the two — e.g. figure_lint_llm.py and section_intro_lint_llm.py need aalto_llm.py but not lintutil.py). Keeping both helpers alongside the scripts covers every linter, so grab both when you download a single script.

Verify what you downloaded before running it¶

How these were built, and the terms of use. These scripts were written with substantial help from large language models (AI coding assistants) and are provided as-is, with no warranty of any kind — you run them at your own risk. They may contain bugs, produce wrong findings, or behave unexpectedly; nothing here is a substitute for your own judgement. Before running them on anything you care about, review the code (it is short and readable by design — see the checks below), and keep backups of your files. The maintainers accept no liability for any loss or damage arising from their use.

These are ordinary Python scripts: running one executes its code with your user account's permissions (your files, your network, any API key you have exported). That is normal for any script you download — but it means you should be sure the files you run are the ones published here and have not been tampered with in transit or on a mirror.

The repository ships a SHA256SUMS file listing the SHA-256 hash of every .py file. After cloning or downloading, verify the scripts match:

cd masterthesis/assets/linters
shasum -a 256 -c SHA256SUMS      # macOS/BSD; prints "<file>: OK" for each
# or, on Linux:
sha256sum -c SHA256SUMS

Every line should end in OK. A FAILED line means that file differs from the published version — stop and re-download from the official source rather than run it. If you fetched a single script with curl, grab the checksum list the same way and check just that file:

curl -O https://ml-theses.org/assets/linters/SHA256SUMS
shasum -a 256 -c SHA256SUMS 2>/dev/null | grep prose_lint.py   # the file you got

The files are short and readable by design. If you want to go further than the checksum, skim them — or scan for the patterns a malicious script would need, none of which appear in this suite:

grep -rnE '\b(eval|exec|os\.system|os\.popen|subprocess.*shell=True|b64decode|__import__|pickle|marshal|socket\.)\b' *.py

For a more thorough, automated pass, run Bandit — the standard security linter for Python. It statically scans for common insecure patterns (the ones the grep above looks for, and many more) and reports each with a severity and confidence rating:

pip install bandit
bandit -r .            # scan every .py in this directory tree

A clean run (no results, or only low-severity informational notes) is a good sign; investigate anything flagged medium or high before running the script. Bandit reads the code without executing it, so it is safe to run on scripts you have not yet trusted.

Maintainer note: regenerate SHA256SUMS after any change to the scripts with shasum -a 256 $(ls *.py | sort) > SHA256SUMS (run from this directory), and commit it in the same change as the edited script.

Setup¶

This is the one-time install. The first line makes a virtual environment (venv) — a private, throwaway Python sandbox for this project, so installing packages here can't disturb the rest of your system. The second line installs the one package most PDF linters need. The third is only for the AI-powered linters.

# one-time setup — the venv sidesteps pip's PEP 668
# "externally managed environment" refusal on Homebrew/Debian Python
python3 -m venv .venv && source .venv/bin/activate
pip install pymupdf          # needed by several PDF linters (see "Needs" column)
export AALTO_API_KEY=...     # *_llm linters only

If you skip the venv and plain pip install pymupdf is refused with an "externally managed environment" error (PEP 668), use pip3 install --user --break-system-packages pymupdf instead — this works on e.g. Aalto's JupyterHub.

Linters whose "Needs" column below is empty read PDFs with either of two PDF-to-text tools — pdftotext (from the poppler package: brew install poppler / apt install poppler-utils) or PyMuPDF — whichever is available. The ones marked PyMuPDF in the table need that specific package; without it they exit with status 2 and print an install hint, so you'll know.

The *_llm linters call the Aalto AI API by default ($AALTO_API_KEY — sign up for a key on the Aalto API developer portal; Aalto network/VPN only — see aalto_llm.py). To send them somewhere else — the Aalto LLM Gateway, a local on-device server, or any OpenAI-style endpoint such as OpenRouter — override the endpoint and model either way:

On the command line, with --base-url (and optionally --model / --vision-model). This works on a single script and on the whole suite:

python3 prose_lint_llm.py thesis.pdf --base-url http://localhost:8080/v1
python3 run_all_linters.py thesis.pdf --llm --base-url http://localhost:8080/v1

Via environment variables — export LLM_BASE_URL (and LLM_MODEL / LLM_VISION_MODEL) once and every linter, run singly or through run_all_linters.py, picks them up. A flag, when given, wins over the matching env var.

Choosing the LLM model¶

You pick the model with --model <id> (or the LLM_MODEL env var); --vision-model / LLM_VISION_MODEL sets the model figure_lint_llm.py uses for figure images. The valid <id> values depend on which endpoint you point at (--base-url / LLM_BASE_URL, above). Leave --model off and each endpoint uses a sensible default.

Which ids are valid changes over time — always confirm against the Aalto sources below. The values here were current in August 2026.

Aalto AI API (the default endpoint) — the OpenAI GPT-5 family, with dated ids. The full catalogue and your subscription key are on the Aalto AI APIs page (Aalto login required):

Model id	Notes
`gpt-5-mini-2025-08-07`	suite default — fast, cheap, strong enough for the linters
`gpt-5-2025-08-07`	most capable; slower and more expensive
`gpt-5-nano-2025-08-07`	smallest/cheapest; use for quick, cheap passes

python3 run_all_linters.py thesis.pdf --llm --model gpt-5-2025-08-07

Aalto LLM Gateway (open-weight models on Aalto hardware; needs the Aalto VPN and a key made at https://llm-gateway.k8s.aalto.fi/). Point --base-url at it, then choose a --model. The authoritative, current list is a one-liner against its /models endpoint:

curl -s https://llm-gateway.k8s.aalto.fi/api/v1/models \
     -H "Authorization: Bearer $AALTO_LLM_KEY" | python3 -m json.tool

As of August 2026 it serves (pass the id verbatim to --model): RedHatAI/gemma-4-31B-it-FP8-Dynamic (Aalto's recommended starter), openai/gpt-oss-120b, Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8, Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 (vision, for --vision-model), Qwen/Qwen3-VL-30B-A3B-Thinking-FP8, Qwen/Qwen3.8-27B-FP8, google/gemma-4-E4B-it, google/codegemma-7b-it. Gateway models "scale to zero": the first request to a cold model can take a few minutes (the client retries the 503s for you). See Local LLM web APIs for details.

export AALTO_LLM_KEY=...     # from https://llm-gateway.k8s.aalto.fi/
python3 run_all_linters.py thesis.pdf --llm \
    --base-url https://llm-gateway.k8s.aalto.fi/api/v1 \
    --model RedHatAI/gemma-4-31B-it-FP8-Dynamic \
    --vision-model Qwen/Qwen3-VL-30B-A3B-Instruct-FP8

(For a local on-device model, see the mlx_lm.server example under Data handling below — same --base-url / --model idea.)

Data handling¶

The *_llm linters send the manuscript text (and, for the figure and caption linters, figure images) to the configured endpoint. A thesis draft is unpublished material, so keep it on an Aalto-hosted gateway:

Default keeps the draft on Aalto infrastructure. The Aalto AI API and the Aalto LLM Gateway run within Aalto's tenant. Per Aalto's own description of these services, inputs are processed under Aalto's agreement and are not used to train the provider's models; this is the route Aalto recommends for unpublished, confidential, or personal material, unlike public services such as ChatGPT. For the authoritative terms, data classification, and GDPR position, check the Aalto AI services and responsible use of AI in research pages — they, not this README, are the source of truth.
A local on-device model is also compliant — and the most private. Point the linters at a server running on your own machine (e.g. mlx_lm.server on Apple Silicon, or Ollama — both expose an OpenAI-style /v1/chat/completions API) with --base-url (or the LLM_BASE_URL env var) — see Setup; either works on a single script or the whole suite. The manuscript never leaves the device, so no network, VPN, or key is needed. The client recognises localhost / 127.0.0.1 as a trusted endpoint and prints no warning. Use a capable model for quality on par with the gateway — a ~14B model such as Qwen3-14B handles the chunked linters (prose, flow, section-intro, caption) well; the whole-thesis linters (thesis_checklist_llm, research_questions_lint_llm, rq_quality_lint_llm, type_consistency_lint_llm) need more context and memory, and figure_lint_llm needs a vision model (Qwen3-VL). Very small models (≤2B) miss real defects — validate before relying on them. Example:
```
mlx_lm.server --model mlx-community/Qwen3-14B-4bit --port 8080 &
export LLM_BASE_URL=http://localhost:8080/v1
export LLM_MODEL=mlx-community/Qwen3-14B-4bit
python3 prose_lint_llm.py thesis.pdf
```
Do not point --base-url at a public endpoint (e.g. OpenRouter) for a real draft — that would send unpublished material to a public AI service, contrary to Aalto's guidance. The client prints a warning when the endpoint is not Aalto-hosted.
Special-category personal data (e.g. interview transcripts, human-subjects data embedded in the draft) needs a data-classification check before linting, even on the Aalto gateway.
When running these on a student's draft, tell the student that supervisor feedback may be produced by routing their manuscript through this suite — the mirror image of the AI-use disclosure asked of them.

Basic usage¶

Run one linter on a compiled thesis PDF (or run everything at once, below):

python3 acronym_lint.py thesis.pdf

== Acronym lint report
File: thesis.pdf

[WARN] NEVER-EXPANDED     p12    'INT4' used 11 time(s) but never expanded (first use at p12).
[WARN] USED-BEFORE-EXPANSION p9  'CNN' used on p9 but expanded only on p17.

Read a finding line left to right — it has four parts. Taking the first line above, [WARN] NEVER-EXPANDED p12 'INT4' used 11 time(s)...:

Severity — how much to worry:
[ERROR] — almost certainly a defect; fix it.
[WARN] — worth reviewing; occasionally a false alarm.
[INFO] — a low-priority heads-up; usually fine, but may ask you to eyeball something (e.g. confirm what a pronoun refers to).
Finding code (NEVER-EXPANDED) — a short, stable name for this kind of problem. Every code is listed, per linter, in the tables further down, so you can look up what it means.
Location (p12) — where in the manuscript: a page number pNN for a PDF, or file:line when you lint LaTeX sources.
Evidence — what was found and why it was flagged, in plain words.

A clean run prints no finding lines and exits with status 0; a run with findings exits 1; a missing file or missing dependency exits 2.

Run everything at once:

python3 run_all_linters.py thesis.pdf              # fast heuristic suite
python3 run_all_linters.py thesis.pdf --llm --bib  # + LLM + bibliography

The first command runs only the fast, rule-based checks. The semantic assessments — research questions, contributions, flow, the thesis checklist and the other *_llm linters in the tables below — require the second form: they run only with --llm.

run_all_linters.py prints each linter's report followed by a one-line per-linter summary (clean / findings / error).

For a visual report instead of scrolling console text, add --dashboard. Want to see what it looks like first? Open the live example dashboard — a real --llm --bib run of this suite on a sample document, rendered exactly as the command below would render your own.

python3 run_all_linters.py thesis.pdf --dashboard          # fast suite only
python3 run_all_linters.py thesis.pdf --llm --bib --dashboard   # full suite

Note that --dashboard changes only how results are shown, not which linters run: the first command renders a dashboard of the fast checks alone. To see the semantic assessment cards (research questions, contributions, flow, checklist) — the ones featured in the example dashboard above — use the second command.

This runs the suite once and, when it finishes, writes a self-contained HTML page — a summary band, one expandable card per linter grouped by theme, and the figure linter's figures × ten-rules matrix as a colour-coded table — then opens it in your default browser. The page is written to <thesis>_lint_dashboard.html in the current directory (override with --dashboard-out FILE); it needs no network and works offline, in light or dark mode. Add --no-open to write the file without launching a browser (headless boxes, CI), or set $BROWSER to choose which browser opens. The console report still streams live while the suite runs, so --dashboard only adds the report — it takes nothing away.

Profiles: thesis vs. research paper¶

The suite targets MSc theses by default, but a thesis and a conference/ journal paper are the same object — an ML manuscript — and ~14 of the linters (prose, acronym, terminology, math, captions, figures, flow, forward-references, citations, unreferenced entities, type-consistency) apply verbatim to either. Pass --profile paper to lint an IEEE/ACM-style paper draft instead (IEEE and ACM are the two main computer-science publishers, and their formatting rules are the norm for CS papers):

python3 run_all_linters.py paper.pdf --profile paper --llm

--profile thesis is the default and changes nothing. --profile paper adapts the suite in three ways:

Skipped (thesis-only): ai_disclosure_lint.py (IEEE/ACM do not mandate a disclosure statement) and thesis_checklist_llm.py.
Swapped in: paper_checklist_llm.py — the reviewer-facing content checklist (contributions, novelty positioning, claims-supported, reproducibility, limitations, and — only for venues that require it — an ethics statement; "venue" means the conference or journal you submit to).
Adapted in place (these five take a --profile flag of their own, so they behave the same run standalone):

Linter	`--profile paper` change
`structure_lint.py`	`LONE-CHILD` downgraded to `INFO` (a two-column paper legitimately has single-subsection sections)
`citation_style_lint.py`	takes `--venue` (default `ieee`); non-IEEE venues skip the IEEE-specific checks rather than misflag them
`data_split_lint_llm.py`	counts only methods the authors themselves train; pretrained/off-the-shelf models used as-is are out of scope
`research_questions_lint_llm.py`	accepts an enumerated contributions list as the unit; drops the thesis-only "revisited in conclusions" penalty
`rq_quality_lint_llm.py`	judges the problem statement + contributions by the same criteria when no explicit research questions are stated

Venue defaults to IEEE. The paper checklist takes --venue neurips|acl|… to require a broader-impact/ethics statement; for ieee/acm that item passes by default.

This applies to the dashboard too: --profile paper --dashboard renders the same HTML report for a paper draft. The HTML is produced by dashboard.py; run_all_linters.py --dashboard calls it for you (see Basic usage). Call dashboard.py directly when you want to render a saved run or embed the page in another document:

python3 dashboard.py thesis.pdf --llm --bib --open        # run, render, open
python3 dashboard.py --from-run saved_run.txt --title "..." --out dash.html
python3 dashboard.py --from-run saved_run.txt --body-only --out body.html

--from-run renders a previously saved run_all_linters.py transcript without re-running anything; --body-only emits just the page markup (a <style> block plus the cards) for embedding in a host that supplies the document shell; --open launches the result in your default browser.

Augmenting a run with an annotated PDF¶

If you have a PDF of your manuscript carrying margin annotations — highlights and typed comments, e.g. from a supervisor's read-through, or your own notes-to-self added in any PDF reader — the suite can use them. Point run_all_linters.py at the annotated PDF with --annotations:

python3 run_all_linters.py thesis.pdf --llm \
    --annotations thesis_annotated.pdf

Two things happen. First, the margin comments are folded into every semantic LLM linter's context, so each one weighs what the human flagged in the area it checks (annotations are advisory — a linter still stays within its own remit). Second, after the run, annotation_coverage_lint_llm.py cross-checks each annotation against everything the suite flagged and reports, per note, whether the automated linters caught the same concern:

COVERED — a linter clearly flags the same issue (it names which one);
PARTIAL — a linter touches the area but misses the specific point;
UNCAUGHT — no linter addresses it (a genuine gap, or simply outside any linter's remit).

It ends with a short list of coverage gaps — classes of concern the suite systematically misses — so an annotated draft doubles as a way to see where the automated checks fall short. Because the coverage check is itself an LLM linter, --annotations is most useful together with --llm.

Under the hood, --annotations accepts either the annotated PDF (it extracts the notes for you via extract_annotations.py) or a pre-extracted annotations JSON. To extract once and reuse:

python3 extract_annotations.py thesis_annotated.pdf --out thesis_annos.json
python3 run_all_linters.py thesis.pdf --llm --annotations thesis_annos.json

The extracted JSON is a plain list of {page, type, comment, quoted} entries and never leaves your machine except as part of the LLM linter calls you opted into with --llm.

Typical workflow for a new thesis PDF¶

python3 run_all_linters.py thesis.pdf              # fast pass, fix mechanics
python3 thesis_checklist_llm.py thesis.pdf         # content checklist
python3 data_split_lint_llm.py thesis.pdf          # per-method train/val/test + diagnosis
python3 research_questions_lint_llm.py thesis.pdf  # RQs answered?
python3 contribution_support_lint_llm.py thesis.pdf # claims backed by evidence?
python3 prose_lint_llm.py thesis.pdf               # deep prose pass
python3 type_consistency_lint_llm.py thesis.pdf    # type/range/dimension of formal claims
python3 bibliography_linter.py thesis.pdf          # verify references

Coverage: ml-theses.org instruction → linter¶

Manuscript preparation¶

Instruction	Linter	How
Problem formulation: data points, features, labels defined	`thesis_checklist_llm.py` `problem_clarity_lint_llm.py`	verdict `problem-formulation` with quoted evidence per defining object of the learning problem (paradigm-aware — e.g. data point / features / label for supervised, state / action / reward / objective for RL): `CLEAR` / `PARTIAL` / `UNCLEAR` / `MISSING`, plus the interface-boundary and unit-identity checks
Abstract readable from elementary (Dictionary) concepts alone	`abstract_selfcontained_lint_llm.py`	grade `GOOD` / `FAIR` / `POOR`; per term `UNDEFINED` / `AMBIGUOUS` / `COMPOUND-JARGON` gaps, each with the naive-reader question and an inline fix
Research scope/questions well-posed (clear, focused, specific, complex, feasible, relevant, self-contained)	`rq_quality_lint_llm.py`	per-question criteria verdicts + scope checks (gap, delimitations, alignment)
Identify data sources and evaluation criteria	`thesis_checklist_llm.py`	verdict `data-sources-eval`
Training loss and validation/test loss explicitly stated	`thesis_checklist_llm.py`	verdict `loss-functions`
Per studied method: training, validation and test set construction described, and the method diagnosed on that split	`data_split_lint_llm.py`	enumerates the trained methods, then per method: `train-set`, `validation-set`, `test-set`, `diagnosis-on-split` verdicts
Numerical results answer the research questions	`research_questions_lint_llm.py` `thesis_checklist_llm.py`	per-question tracing global verdict `results-discussed`
Each claimed contribution is backed by a result (theorem, experiment, analysis) that actually supports it	`contribution_support_lint_llm.py`	per claim: `SUPPORTED` / `PARTIAL` / `UNSUPPORTED` / `ASSERTED` with the backing result located and a why/gap
Stated contribution faithfully presented (not over- or under-sold)	`contribution_faithfulness_lint_llm.py`	actual main contribution vs. the claim: `FAITHFUL` / `OVERSTATED` / `UNDERSTATED` / `MISALIGNED` / `UNCLEAR`
Use appropriate baselines or benchmarks	`thesis_checklist_llm.py`	verdict `baselines`
Chapter/section introductions	`section_intro_lint_llm.py` `thesis_checklist_llm.py` `prose_lint_llm.py`	intro maps its subsections verdict `section-intros` `unmotivated-section`
Reference all numbered equations using `\eqref{}`	`math_typeset_lint.py`	`REF-NOT-EQREF` (LaTeX)
All numbered equations, tables, figures referenced in the text	`unreferenced_entity_linter.py`	`UNREFERENCED`, `UNLABELED-EQ`, `UNLABELED-FLOAT` (LaTeX + PDF)
Present new methods as pseudocode	`thesis_checklist_llm.py`	verdict `pseudocode`
Model diagnosis via numerical experiments and mathematical analysis	`thesis_checklist_llm.py`	verdict `model-diagnosis`
Figures clear, labelled, informative captions	`figure_lint_llm.py` `caption_lint.py` `caption_lint_llm.py` `thesis_checklist_llm.py`	rendered figures scored against the PLOS Ten Simple Rules (figures × rules matrix; pixels + vision LLM) `SHORT-CAPTION`, `NO-CAPTION` `WEAK-CAPTION` (per-caption LLM) verdict `captions-informative`
References formatted per IEEE guidelines	`citation_style_lint.py`	style/entry/citation checks (LaTeX + PDF)
Terms from the Aalto Dictionary of ML	`terminology_lint.py`	`NON-DICTIONARY`, `TERM-MIX` (dictionary term first per cluster)
Central concepts given a source (provenance of the load-bearing ideas)	`central_concept_citation_lint_llm.py`	per central concept: `CITED` / `OWN-COINAGE` / `ELEMENTARY` / `UNCITED` / `ATTR-VAGUE`, with the fix
Every chapter/section has zero or >= 2 subdivisions	`structure_lint.py`	`LONE-CHILD` (LaTeX + PDF)

Suite self-check¶

Purpose	Linter	How
Does the automated suite catch what a human reviewer flagged? (find blind spots)	`annotation_coverage_lint_llm.py`	cross-checks the reviewer's PDF margin annotations against the suite's own output: `COVERED` / `PARTIAL` / `UNCAUGHT`

Typesetting mathematical texts¶

Instruction	Linter	How
Inline math for short expressions; display math for central/referenced equations	`math_typeset_lint.py`	`LONG-INLINE` (+ `EQNARRAY` hygiene)
Punctuate displayed equations as part of the sentence	`math_typeset_lint.py`	`EQ-NO-PUNCT`, `EQ-PUNCT-CHECK`

Self-editing pass (prose linter)¶

Instruction	Linter	How
Excessive forward referencing	`prose_lint.py` `crossref_forward_lint.py` `forward_ref_lint.py` `forward_ref_lint_llm.py`	`FORWARD-CUE` phrases floats (figures/tables) defined pages later concepts used before defined (regex and LLM variants)
Undefined or re-defined acronyms	`acronym_lint.py`	`USED-BEFORE-EXPANSION`, `NEVER-EXPANDED`, `RE-EXPANDED`
Inconsistent terminology / synonym switching	`terminology_lint.py` `prose_lint_llm.py`	`TERM-MIX` `synonym-switch`
Vague quantifiers without a number	`prose_lint.py` `prose_lint_llm.py`	`VAGUE-QUANTIFIER` `vague-quantifier`
Jargon / undefined evaluative claims ("smoothest convergence")	`prose_lint.py` `prose_lint_llm.py`	`JARGON` `jargon`
Uncited claims	`prose_lint_llm.py` `bibliography_linter.py`	`uncited-claim` verifies the cited references
Dangling references ("this shows" without antecedent)	`prose_lint.py` `prose_lint_llm.py`	`DANGLING-REFERENCE` `dangling-reference`
Unmotivated sections	`prose_lint_llm.py` `thesis_checklist_llm.py`	`unmotivated-section` verdict `section-intros`
Tense drift	`prose_lint_llm.py` (`tense-drift`)	LLM
Broken idioms ("corner cuttings on safety")	`prose_lint_llm.py` (`broken-idiom`)	LLM
Informal register ("a bunch of", contractions)	`prose_lint_llm.py` (`informal-register`)	LLM
Category errors (an algorithm named where a metric is meant)	`prose_lint_llm.py` (`category-error`)	LLM
Empty buzzwords ("framework", "leverage", unearned "robust")	`prose_lint_llm.py` (`empty-buzzword`)	LLM
Section openers stand alone; no narrative jumps between paragraphs	`flow_lint_llm.py`	`OPAQUE-OPENER` (judged without the preceding text), `FLOW-BREAK`
Type / range / dimension mismatches in formal claims	`type_consistency_lint_llm.py`	`TYPE-MISMATCH`, `RANGE`, `DIMENSION`, `BRIDGE-LOOSE`

Responsible use of AI, references quality¶

Instruction	Linter	How
Disclose AI use in a dedicated statement (not in Methods)	`ai_disclosure_lint.py`	`NO-AI-STATEMENT`, `IN-METHODS`
Record the tool, version, and settings	`ai_disclosure_lint.py`	`NO-TOOL-NAMED`, `NO-VERSION`
Citations verified; high-quality references	`bibliography_linter.py`	existence + author/title/venue vs Crossref/arXiv/DBLP; `PREPRINT`, `WEB-SOURCE`, `NOT-FOUND`

Not machine-checkable (process instructions)¶

Some instructions concern how you work, not the finished manuscript, so no linter can check them:

The writing process itself — a linter only sees the compiled PDF, so it cannot tell whether an expected section is missing, nor in what order the chapters were written.
Keeping a lab notebook and budgeting enough revision rounds.
Not uploading confidential data to public AI services.
Accountability for the content — the text and its claims remain yours.

The closest proxy: run the suite before every revision round.

The linters¶

The full catalogue, one row per linter. Input is the file type it accepts (.pdf for a compiled PDF, .tex for LaTeX source, .bib for a bibliography file). Needs is any extra requirement beyond plain Python: network (internet access), PyMuPDF (the pip install pymupdf package), or Aalto AI API (an LLM linter — needs AALTO_API_KEY); a dash means no extras. A short prose note on each linter follows the table.

Script	Checks	Input	Needs
`bibliography_linter.py`	cited references exist; author/title/ venue/year match Crossref/arXiv/DBLP	`.bib`, `.pdf`	network
`structure_lint.py`	sectioning units with exactly one subdivision	`.tex`, `.pdf`	—
`unreferenced_entity_linter.py`	numbered equations/tables/figures never referenced	`.tex`, `.pdf`	—
`crossref_forward_lint.py`	references to floats (figures, tables, algorithms) defined many pages later	`.pdf`	PyMuPDF
`forward_ref_lint.py`	concepts used before defined (regex)	`.pdf`	PyMuPDF
`forward_ref_lint_llm.py`	concepts used before defined (LLM)	`.pdf`	PyMuPDF + Aalto AI API
`acronym_lint.py`	acronym expanded at first use, no re-expansion	`.tex`, `.pdf`	—
`prose_lint.py`	vague quantifiers, dangling refs, forward cues	`.tex`, `.pdf`	—
`unresolved_reference_lint.py`	uncited appeals to companion/forthcoming studies; label-code schemes (R1, T6, …) used without a definition in the text	`.tex`, `.pdf`	—
`terminology_lint.py`	synonym mixing vs Aalto Dictionary terms	`.tex`, `.pdf`	—
`math_typeset_lint.py`	display-math punctuation, `\eqref`, long inline math	`.tex`	—
`citation_style_lint.py`	IEEE reference/citation format	`.tex`, `.pdf`	—
`caption_lint.py`	missing/too-short figure & table captions	`.tex`, `.pdf`	—
`caption_lint_llm.py`	per-caption quality: states what's shown, defines quantities, self-contained, sentence form	`.tex`, `.pdf`	Aalto AI API
`ai_disclosure_lint.py`	dedicated AI-use statement with tool + version	`.tex`, `.pdf`	—
`thesis_checklist_llm.py`	9-item manuscript checklist, PASS/FAIL + evidence (thesis profile)	`.pdf`	Aalto AI API
`paper_checklist_llm.py`	reviewer content checklist for a research paper, PASS/FAIL + evidence (paper profile; venue-gated ethics item)	`.pdf`	Aalto AI API
`related_work_faithfulness_llm.py`	finds the <=3 most-related works and checks the draft represents them faithfully against their real abstracts	`.pdf`	Aalto AI API + OpenAlex
`data_split_lint_llm.py`	per studied ML method: train/validation/test set construction and diagnosis on that split	`.pdf`	Aalto AI API
`research_questions_lint_llm.py`	each stated research question: answered? where? on what evidence?	`.pdf`	Aalto AI API
`rq_quality_lint_llm.py`	how well-posed are research questions and scope (university criteria)?	`.pdf`	Aalto AI API
`contribution_support_lint_llm.py`	per claimed contribution: which result (theorem/proof, experiment, analysis) backs it, and does it?	`.pdf`	Aalto AI API
`contribution_faithfulness_lint_llm.py`	one holistic verdict: is the headline contribution over- or under-sold?	`.pdf`	Aalto AI API
`abstract_selfcontained_lint_llm.py`	is the abstract self-contained from elementary (Aalto Dictionary) concepts? (paper profile)	`.pdf`	Aalto AI API
`central_concept_citation_lint_llm.py`	are the paper's central concepts sourced (cited / own-coinage / elementary / uncited)? (paper profile)	`.pdf`	Aalto AI API
`problem_clarity_lint_llm.py`	is the learning problem stated clearly? Identifies the paradigm (supervised, unsupervised, RL, generative) and grades its defining objects — e.g. data point / features / label, or state / action / reward / objective (paper profile)	`.pdf`	Aalto AI API
`annotation_coverage_lint_llm.py`	cross-checks reviewer PDF annotations against what the suite flagged (`COVERED`/`PARTIAL`/`UNCAUGHT`)	run output + annotations	Aalto AI API
`figure_lint_llm.py`	figures scored against the PLOS Ten Simple Rules for Better Figures (figures × ten-rules matrix)	`.pdf`	PyMuPDF (+ Aalto AI API unless `--no-llm`)
`section_intro_lint_llm.py`	does each chapter/section intro map its subsections and tie them together?	`.pdf`	PyMuPDF + Aalto AI API
`type_consistency_lint_llm.py`	formal claims well-typed: relations over same-type operands, values in range, dimensionless quantities unit-free (`TYPE-MISMATCH`, `RANGE`, `DIMENSION`, `BRIDGE-LOOSE`)	`.pdf`	Aalto AI API
`flow_lint_llm.py`	narrative flow: section openers that stand alone, no paragraph-to-paragraph discontinuities	`.pdf`	PyMuPDF + Aalto AI API
`prose_lint_llm.py`	LLM self-editing pass (uncited claims, tense drift, jargon, …)	`.tex`, `.pdf`	Aalto AI API
`run_all_linters.py`	runs everything above; `--dashboard` renders + opens an HTML report; `--annotations` folds in reviewer PDF notes	either	—
`dashboard.py`	renders a run as a self-contained HTML dashboard (`--open` to launch in a browser)	either	—

Shared modules: lintutil.py (text extraction, report format), aalto_llm.py (Aalto AI API client; also used by the *_llm linters). extract_annotations.py (a helper, not a linter) pulls a PDF's margin highlights and comments into a JSON list — see Augmenting a run with an annotated PDF.

Maintainer note: the linter scripts in this directory are published from a single upstream linter suite and verified against the shipped SHA256SUMS. Do not hand-edit an individual linter here — the next sync overwrites it, and until then the change fails the site build's SHA256SUMS gate. Edit upstream and re-sync. The non-.py assets (this README.md, the demo dashboard) are maintained in this repo. The upstream suite ships one extra reviewer-only linter (own_work_relation_lint_llm.py) that is intentionally not published here.

Notes on individual linters¶

bibliography_linter.py — verifies that cited references exist and match a real record. Findings include NOT-FOUND (possibly hallucinated), AUTHOR-MISMATCH, TITLE-DRIFT, VENUE-MISMATCH, PREPRINT, YEAR-MISMATCH. Results cached in .bibcheck_cache.json.

unreferenced_entity_linter.py — LaTeX mode flags \labels no \ref/\eqref/\cref points to, numbered math without \label, captioned floats without \label; PDF mode works from captions and right-aligned equation numbers.

structure_lint.py — a lone subdivision cannot articulate a division: LONE-CHILD flags a chapter with a single section, a section with a single subsection, etc. PDF mode prefers the embedded bookmark outline (PyMuPDF) and falls back to scanning extracted text for numbered headings; LaTeX mode parses the sectioning commands (starred variants are unnumbered and skipped).

crossref_forward_lint.py — a float is a figure, table, or algorithm, which LaTeX positions ("floats") wherever it fits rather than exactly where you place it. This linter flags "as depicted in Figure 7" on page 3 when Figure 7 appears on page 21 (threshold: > 1 page forward by default).

forward_ref_lint.py / forward_ref_lint_llm.py — paragraph-level conceptual forward references; the LLM version maintains a running introduced-concept set, caches per paragraph (<input>.fwdref_cache.jsonl, --resume).

terminology_lint.py — each synonym cluster lists the Aalto Dictionary of ML term first; TERM-MIX fires only when two or more variants each occur --min-count times.

thesis_checklist_llm.py — one LLM call over the full extracted text (page markers included) returns PASS/FAIL/UNCLEAR per checklist item with a quoted, page-referenced evidence snippet and a concrete fix for each FAIL.

unresolved_reference_lint.py — catches two pointers that send the reader to something they cannot inspect, and that slip past the other linters. COMPANION-REF: an uncited appeal to a companion / separate / forthcoming / related study that carries part of the argument (e.g. "the remaining nine are evaluated in companion studies") — cited sentences ([n] / \cite) are not flagged, since the reader can then find the work. UNDEFINED-CODE: a scheme of short label codes (R1, T6, RQ2, …) used as load-bearing shorthand but never defined; a code counts as defined when it appears with a gloss ("R1 (Fault Containment)", "R1: …", "R1 — …") or a defining keyword, while a bare range "(R1–R5)" does not define its members. To stay precise the code check requires a prefix to have ≥ 2 distinct small-numbered members and to sit near a scheme keyword (criteria/requirement/research question/…) or have ≥ --min-members members, so the "L1"/"L2" norms, an "R2" (R-squared) or "F1" score, an "S3" bucket, or a "CO2" reading are not mistaken for a scheme. The forward-reference linters model concepts defined LATER IN THE SAME document, not appeals to external papers, so this check is complementary.

related_work_faithfulness_llm.py — goes beyond checking that a related work is cited (bibliography_linter.py) or that novelty is asserted (paper_checklist_llm.py's novelty-positioned) to check whether the draft represents its prior work faithfully. Three stages: (1) an LLM reads the draft and picks the ≤3 cited works it treats as most related, extracting how the draft positions each and the delta it claims; (2) each work's real abstract is fetched from OpenAlex by title (only the reference title leaves the machine — the same class of query the bibliography linter makes); (3) an LLM compares the draft's stated relation to what the abstract actually shows. Findings: RELATION-MISSTATED, NOVELTY-OVERSTATED, SHALLOW-POSITIONING, UNVERIFIABLE, and (info) FAITHFUL. --discover external additionally queries OpenAlex with the draft's own title (still title-only) to flag a clearly related work that is not cited (RELATED-WORK-OMITTED). Manuscript text goes only to the LLM endpoint (Aalto by default); only titles reach OpenAlex.

paper_checklist_llm.py — the --profile paper analogue of thesis_checklist_llm.py, same one-call PASS/FAIL/UNCLEAR machinery but a reviewer-facing rubric: contribution-stated, problem-formulation, novelty-positioned, claims-supported, baselines, results-answer-claims, reproducibility, limitations, and a venue-conditional ethics-impact. --venue (default ieee) governs the ethics item: it is only required for neurips/acl/emnlp/iclr and passes by default otherwise. The prompt tells the model to judge by conference-reviewer standards (sections not chapters, contributions not research-question chapters, page-limited exposition), so a terse but complete treatment passes.

data_split_lint_llm.py — where thesis_checklist_llm.py judges the manuscript globally (one loss-functions / model-diagnosis verdict for the whole thesis), this linter works per studied ML method. One LLM call first enumerates the methods the thesis actually trains (methods only named in related work are excluded), then for each emits train-set, validation-set, test-set, and diagnosis-on-split verdicts (PASS/FAIL/UNCLEAR) with page-referenced evidence and a fix for each FAIL. validation-set also passes when a method legitimately needs no held-out validation set and the thesis says so.

prose_lint_llm.py — chunked LLM pass (--chunk-chars, --concurrency, --checks to select categories, --pages for a partial run).

caption_lint_llm.py — judges every figure/table caption against Rule 4 ("Captions Are Not Optional") of the PLOS "Ten Simple Rules for Better Figures": states what's shown, defines the quantities/symbols it mentions, self-contained for a figure-skimming reader, proper sentence form (a caption like "wibson protocol visualized" fails on all four). One WEAK-CAPTION per deficient caption with the violated criteria and a suggested rewrite. Uses the full GPT-5 model by default; --match 4 restricts to one figure, --per-batch/--concurrency tune the calls.

figure_lint_llm.py — locates every figure via its caption, renders the figure region at 144 dpi, and scores it against the ten rules of Rougier, Droettboom & Bourne, "Ten Simple Rules for Better Figures" (PLOS Comput Biol 2014). The output is a matrix: one row per figure, one column per rule (R1 know your audience, R2 identify your message, R3 adapt to the medium, R4 captions are not optional, R5 do not trust the defaults, R6 use color effectively, R7 do not mislead the reader, R8 avoid chartjunk, R9 message trumps beauty, R10 get the right tool). A vision model gives each cell a verdict — ✓ pass, ~ minor issue, ✗ clear violation, · not applicable — with the body-text font size as the legibility yardstick, and a per-figure notes section explains every flagged cell. Pixel heuristics (computed offline with PyMuPDF) feed the relevant rules as measurements: the blank-background fraction above --max-white informs R8, and an embedded raster below --min-dpi at printed size informs R3; with --no-llm only these heuristic cells are filled and the rest are left unassessed (?). A near-empty rendered region (≥98.5% blank) is reported as a mislocated figure, not scored as blank. --save-crops DIR writes the judged renderings for inspection; --figures 3,7 restricts the run. On the Aalto LLM Gateway the Qwen3-VL vision model is used automatically.

section_intro_lint_llm.py — for every chapter/section that has subsections (outline from the PDF's embedded TOC), extracts the text between the heading and the first subsection heading and judges it: GOOD only if each subsection is framed as an upcoming part (explicit section reference, ordinal enumeration, or forward-pointing phrasing) with its content indicated, plus a connective thread. A thematic categorization that never says which subsection treats which theme is judged WEAK — the alignment exists only in hindsight. Uses the full GPT-5 model by default (the mini tier lets near-miss intros pass). --match 2.4 restricts to one unit; --levels 1,2 selects outline depths.

flow_lint_llm.py — two checks on the manuscript as a narrative. opener: the first sentence(s) after every chapter/section heading are judged WITHOUT the preceding text — exactly like a reader entering at the heading — and flagged OPAQUE-OPENER when they hang on an antecedent across the heading (a bare pronoun, or a definite noun phrase like "A difference of almost an order of magnitude illustrates ..."); named references ("as shown in Section 2.3") and deixis to the unit itself are fine. Judged with the full GPT-5 model (one call per heading, --levels 1,2,3). flow: overlapping windows of consecutive paragraphs are scanned for non-sequitur transitions and paragraphs that presuppose not-yet-introduced material (FLOW-BREAK); headings legitimately reset the narrative and float furniture is skipped. Complements section_intro_lint_llm.py (which judges only units WITH subsections) and the dangling-reference checks (which only see pronoun anaphora, and judge resolvability in-chunk rather than from the heading).

rq_quality_lint_llm.py — judges how well-posed the research questions and scope are, against criteria compiled from authoritative university guidance (the Monash University Library research-question checklist — focused, researchable, feasible, specific, complex, relevant; the George Mason University Writing Center research-question criteria — clear, focused, concise, complex, arguable; and the FINER framework), plus a self-containment check (every technical term in the question is actually defined — not merely mentioned — at or before the page where the question is stated). Per question: STRONG/ADEQUATE/WEAK with the violated criteria named and a concrete reformulation (a yes/no-formed engineering question with obvious quantitative intent is treated as a minor form issue, not a defect). Scope-level checks: gap identified, delimitations stated, question– objective alignment, aim coverage. Uses the full GPT-5 model by default.

research_questions_lint_llm.py — extracts every explicitly stated research question (RQ lists, hypotheses, numbered objectives) and judges each one: ANSWERED / PARTIALLY-ANSWERED / UNANSWERED, where the answer is developed, whether the presented results actually support it, and whether the conclusions chapter revisits it. Warns on NO-RQS (none stated) and NOT-REVISITED. Exit 1 unless every question is answered and revisited.

contribution_support_lint_llm.py — one LLM call over the full text builds the support chain for each claimed contribution: the claim, its TYPE (theoretical / empirical / analysis / methodological / dataset), the BACKING result the manuscript offers (theorem+proof / experiment / analysis / ablation / construction / dataset / none) with its LOCATION (Theorem 3.2 (p5), Table 2 (p7)), and a support level — SUPPORTED, PARTIAL, UNSUPPORTED, or ASSERTED — with a one-line why (how the result entails the claim) or gap (the exact shortfall). Theorems get special scrutiny: a formal claim is SUPPORTED only when the theorem's statement and assumptions actually entail the prose claim and a proof is given — a theorem narrower or weaker than the sentence it backs is a PARTIAL "theorem-claim gap", and a formal-sounding claim proved only by an informal argument does not pass. Where research_questions_lint_llm.py traces stated questions, this traces stated contributions to the evidence meant to establish them. Exit 1 unless every claim is SUPPORTED.