pg.ddx.io LLM-written bug reports

DDX for PostgreSQL — LLM-written bug reports

HOW MANY BUG REPORTS ARE WRITTEN WITH AN LLM?

Bug reports reach the project through the form on postgresql.org, which posts each one to pgsql-bugs as a BUG # message. This page estimates, for each year since 2019, what share of those reports were written with a large language model. It is an estimate for the whole year’s reports, not a judgement of any one report or person: no per-report or per-person value is computed for publication, and none appears anywhere on this site.

Through 2024 the estimate sits at or below what the method can tell apart from zero. It rises in 2025, and 2026 is well above every earlier year. Take the 2026 figure as “well above” rather than as a precise share: the reference set of known LLM-written reports is small, and most of it is from 2026 itself.

Estimated share of bug reports written by an LLM, by year 2019: 1% (95% interval 0% to 4%); 2020: 0% (95% interval 0% to 3%); 2021: 2% (95% interval 0% to 6%); 2022: 3% (95% interval 0% to 7%); 2023: 5% (95% interval 1% to 9%); 2024: 6% (95% interval 2% to 9%); 2025: 11% (95% interval 6% to 17%); 2026: 59% (95% interval 50% to 67%) 0% 25% 50% 75% 100% method floor 2019 2020 2021 2022 2023 2024 2025 2026
YearBug reportsEstimated LLM share95% intervalSecond reader, at least
20196001%0%–4%0% of 60
20205380%0%–3%2% of 60
20214742%0%–6%0% of 60
20223233%0%–7%0% of 60
20234565%1%–9%0% of 60
20244336%2%–9%3% of 90
202536111%6%–17%8% of 120
202648759%50%–67%15% of 120

Bars are the estimate; whiskers are its 95% interval. The dashed line is the method floor: the top of the 2019–2021 intervals, the years used as the human reference, so a bar below it is not distinguishable from none. 2026 covers reports through 21 November 2026. Data: llm-bug-reports.json.

HOW IT IS ESTIMATED

The estimate does not classify reports one by one. It follows the population method of Liang et al. (2024, arXiv 2403.07183): words appear at different rates in human-written and LLM-written text, so the word rates of a whole year’s reports are fitted as a mixture of the two, and the mixing weight is the share. The human reference is the reports of 2019–2021. The LLM reference is a set of reports found to be LLM-written, either by a person reading them or by a second model; those reports are left out of the fit and added back as known LLM-written.

As a control, ordinary replies on pgsql-bugs (not bug reports) stay between 0 and 2 percent through 2025.

LIMITS

The method sees word choice, so it detects text that reads like an LLM’s. A report that was drafted with an LLM and then rewritten by its author looks human to it, and a person who happens to write in that style looks LLM-assisted. The estimate will be revised as the reference sets grow; this page shows the newest one. LLM help with a bug report is not wrong, and nothing here suggests it is: the point is to know how the project’s incoming reports are changing.