# agents.txt — pg.ddx.io
#
# For AI agents, crawlers, and the people who operate them.
# Humans: see https://pg.ddx.io/ for the same guidance with worked examples.

## Who we are

A free, read-only, public mirror of the PostgreSQL community archives since
1991: every mailing list, the full git history of postgres and its ecosystem,
commitfest, buildfarm, docs and wiki -- cross-linked, so a commit leads to the
thread that argued for it and a thread leads to the code it produced.

Agents and crawlers are WELCOME. There is no login, no paywall, no API key.

## Please use the road that fits your job

This site runs on ONE server. The same corpus can be fetched in one cheap
transfer or in millions of expensive ones. Please pick the cheap one.

### You want to ANSWER A QUESTION about PostgreSQL
Use the MCP server. It is built for exactly this.

    Endpoint:   https://pg.ddx.io/mcp   (MCP Streamable HTTP, JSON-RPC 2.0)
    Tools:      https://pg.ddx.io/mcp-tools   (every tool, arguments, examples)

A good loop, and what each step costs:
  1. hybrid_search {"query": "..."}          ranked hits WITH an excerpt of each
                                             (~2 KB for 3 hits). No need to fetch
                                             a hit to judge it.
  2. get_message   {"message_id": "..."}     one message in full (pass
                                             body_only: true for just the text)
  3. get_thread    {"message_id": "...",     an outline of the whole thread,
                    "include_bodies": false} then fetch only what you need
  4. git_search    {"query": "..."}          commits; each carries its
                                             Discussion: trailer back to the thread
  5. commit_history {"commit_id": "..."}     commit -> discussion -> commitfest

Omit `inbox` and it defaults to pgsql-hackers; omit `repository` and it
defaults to postgres. `repo` and `repository` are accepted interchangeably.

get_thread is PAGED (20 messages, bodies capped at 4000 chars). An unpaged
thread with bodies can exceed 1 MB -- request `limit: 500, max_body_chars: 0`
only if you truly need every byte.

### You want a COPY of a mailing list (training, indexing, research)
Clone it. Do not crawl it.

    git clone https://pg.ddx.io/m/pgsql-hackers.git
    git -C pgsql-hackers fetch                     # later: only the new mail

Every list is a public-inbox v2 git archive; the list of lists is at
https://pg.ddx.io/archive. Message ranges also come as gzipped mbox:
https://pg.ddx.io/m/<list>/<from>-<to>.mbox.gz  (numbers: /m/<list>/info.json)

### You want to KNOW WHEN something new arrives
Poll a feed, not the pages.

    New mail:     https://pg.ddx.io/m/<list>/new.atom   (or rss.xml)
    New threads:  https://pg.ddx.io/m/<list>/topics_new.atom
    New commits:  https://pg.ddx.io/gitweb/?p=postgres.git;a=atom
    From an MCP client: get_new_messages {"after": <last article_num seen>}
    From a mail client: IMAP IDLE on imap.pg.ddx.io:993, newest slice folder

## Please do NOT

  * Fetch messages one at a time over HTTP (/m/<list>/<message-id>/raw and its
    HTML page) to build a copy of a list. On 2026-09-24 one crawler made 364,354
    /m/ requests in a single day, 134,510 of them single-message /raw fetches --
    the same data one `git clone` delivers. Traffic like that is what caused the
    outage of that night.
  * Walk every thread page (/t/, /T/) to discover messages. The git archive and
    the feeds already enumerate them.
  * Open many parallel connections. Two is plenty; honour Crawl-delay: 2.

None of this is enforced by refusal. We would rather serve you than block you.
Heavy clients are rate-limited at the edge and receive HTTP 429, which means
"slow down", not "go away".

## Identify yourself

Send a User-Agent that names your crawler and a URL or address we can reach.
If you need something we do not offer -- a bulk snapshot, a new export format,
a higher rate for a research project -- ask: https://pg.ddx.io/contact

## Also here

  /llms.txt      concise index for language models
  /robots.txt    crawl rules (everything allowed; Crawl-delay: 2)
  /openapi.yaml  REST API contract
  /status        live health, ingestion freshness