DDX for PostgreSQL — LLM-written bug reports
HOW MANY BUG REPORTS ARE WRITTEN WITH AN LLM?
Bug reports reach the project through the form on postgresql.org, which posts each
one to pgsql-bugs as a BUG # message. This page estimates, for each year
since 2019, what share of those reports were written with a large language model. It
is an estimate for the whole year’s reports, not a judgement of any one report
or person: no per-report or per-person value is computed for publication, and none
appears anywhere on this site.
Through 2024 the estimate sits at or below what the method can tell apart from zero. It rises in 2025, and 2026 is well above every earlier year. Take the 2026 figure as “well above” rather than as a precise share: the reference set of known LLM-written reports is small, and most of it is from 2026 itself.
| Year | Bug reports | Estimated LLM share | 95% interval | Second reader, at least |
|---|---|---|---|---|
| 2019 | 600 | 1% | 0%–4% | 0% of 60 |
| 2020 | 538 | 0% | 0%–3% | 2% of 60 |
| 2021 | 474 | 2% | 0%–6% | 0% of 60 |
| 2022 | 323 | 3% | 0%–7% | 0% of 60 |
| 2023 | 456 | 5% | 1%–9% | 0% of 60 |
| 2024 | 433 | 6% | 2%–9% | 3% of 90 |
| 2025 | 361 | 11% | 6%–17% | 8% of 120 |
| 2026 | 487 | 59% | 50%–67% | 15% of 120 |
Bars are the estimate; whiskers are its 95% interval. The dashed line is the method floor: the top of the 2019–2021 intervals, the years used as the human reference, so a bar below it is not distinguishable from none. 2026 covers reports through 21 November 2026. Data: llm-bug-reports.json.
HOW IT IS ESTIMATED
The estimate does not classify reports one by one. It follows the population method of Liang et al. (2024, arXiv 2403.07183): words appear at different rates in human-written and LLM-written text, so the word rates of a whole year’s reports are fitted as a mixture of the two, and the mixing weight is the share. The human reference is the reports of 2019–2021. The LLM reference is a set of reports found to be LLM-written, either by a person reading them or by a second model; those reports are left out of the fit and added back as known LLM-written.
- Each reporter counts at most three reports a year, so a few prolific reporters cannot move a year on their own. Without that cap, one reporter who filed an eighth of 2023’s reports made that year look higher than 2024.
- The interval comes from resampling reporters, not reports.
- The estimate is calibrated: mixing known shares of 0, 30 and 90 percent into held-out text and fitting them gave 6, 28 and 71 percent before calibration.
- Second reader: a language model running on our own machine read a sample from each year (the last column of the table). It rarely says “LLM”, so its share is a lower bound. It rises in the same years as the estimate.
- Non-native English is not treated as a sign of LLM use: on held-out mail, the model flagged none of the messages from non-native writers.
As a control, ordinary replies on pgsql-bugs (not bug reports) stay between 0 and 2 percent through 2025.
LIMITS
The method sees word choice, so it detects text that reads like an LLM’s. A report that was drafted with an LLM and then rewritten by its author looks human to it, and a person who happens to write in that style looks LLM-assisted. The estimate will be revised as the reference sets grow; this page shows the newest one. LLM help with a bug report is not wrong, and nothing here suggests it is: the point is to know how the project’s incoming reports are changing.