pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: imchifan8@gmail.com
Subject: BUG #19886: pg_terminate_backend timeout expires early after latch wakeups
Date: Wed, 11 Nov 2026 14:49:13 +0000
Message-ID: <19886-16f4759c592d0c00@postgresql.org> (raw)

The following bug has been logged on the website:

Bug reference:      19886
Logged by:          Noah
Email address:      imchifan8@gmail.com
PostgreSQL version: 18.6
Operating system:   Linux/amd64
Description:        

pg_terminate_backend(pid, timeout) can return false and warn that the full
timeout elapsed after only a small fraction of that duration. This occurs
when the target remains alive after SIGTERM and unrelated events repeatedly
wake the waiting backend's latch.

The timeout should measure elapsed wall-clock time. Early latch wakeups
should not consume portions of the timeout that were not actually waited.

How to repeat:

This requires a Linux PostgreSQL instance with dblink available and a role
permitted to create the extension, signal backends, and execute COPY TO
PROGRAM.

Run the following as `psql -X -f reproduce.sql`:

```
\set ON_ERROR_STOP on

SELECT EXISTS (
  SELECT 1 FROM pg_extension WHERE extname = 'dblink'
) AS dblink_preexisting \gset

CREATE EXTENSION IF NOT EXISTS dblink;

CREATE TEMP TABLE timeout_result (
  control_result boolean,
  control_elapsed_ms numeric,
  signaled_result boolean,
  signaled_elapsed_ms numeric,
  reproduced boolean
);

DO $test$
DECLARE
  control_pid integer;
  test_pid integer;
  control_started timestamptz;
  test_started timestamptz;
  control_elapsed numeric;
  test_elapsed numeric;
  control_result boolean;
  test_result boolean;
BEGIN
  /* A stopped backend cannot process SIGTERM, so this must time out. */
  PERFORM dblink_connect(
    'control',
    'dbname=' || current_database() ||
    ' application_name=terminate_control');
  PERFORM pg_stat_clear_snapshot();
  SELECT pid INTO STRICT control_pid
    FROM pg_stat_activity
   WHERE application_name = 'terminate_control';
  PERFORM dblink_send_query(
    'control',
    $q$COPY (SELECT pg_backend_pid()) TO PROGRAM
       'read pid; kill -STOP "$pid"'$q$);
  PERFORM pg_sleep(0.25);

  control_started := clock_timestamp();
  control_result := pg_terminate_backend(control_pid, 1000);
  control_elapsed := 1000 * extract(epoch FROM
                                     clock_timestamp() - control_started);

  EXECUTE format(
    $q$COPY (SELECT %s) TO PROGRAM 'read pid; kill -CONT "$pid"'$q$,
    control_pid);
  PERFORM pg_sleep(0.20);

  BEGIN
    PERFORM dblink_disconnect('control');
  EXCEPTION WHEN OTHERS THEN
    NULL;
  END;

  /* Repeated SIGUSR1 delivery wakes this backend's latch. */
  PERFORM dblink_connect(
    'target',
    'dbname=' || current_database() ||
    ' application_name=terminate_target');
  PERFORM pg_stat_clear_snapshot();
  SELECT pid INTO STRICT test_pid
    FROM pg_stat_activity
   WHERE application_name = 'terminate_target';
  PERFORM dblink_send_query(
    'target',
    $q$COPY (SELECT pg_backend_pid()) TO PROGRAM
       'read pid; kill -STOP "$pid"'$q$);
  PERFORM pg_sleep(0.25);

  EXECUTE $q$COPY (SELECT pg_backend_pid()) TO PROGRAM
    'read pid; nohup sh -c ''for i in $(seq 1 80); do kill -USR1 "$1"; sleep
0.005; done'' sh "$pid" >/dev/null 2>&1 &'$q$;

  test_started := clock_timestamp();
  test_result := pg_terminate_backend(test_pid, 3000);
  test_elapsed := 1000 * extract(epoch FROM
                                  clock_timestamp() - test_started);

  EXECUTE format(
    $q$COPY (SELECT %s) TO PROGRAM 'read pid; kill -CONT "$pid"'$q$,
    test_pid);
  PERFORM pg_sleep(0.60);

  BEGIN
    PERFORM dblink_disconnect('target');
  EXCEPTION WHEN OTHERS THEN
    NULL;
  END;

  INSERT INTO timeout_result
  VALUES (control_result, control_elapsed, test_result, test_elapsed,
          control_result IS FALSE
          AND control_elapsed >= 800
          AND test_result IS FALSE
          AND test_elapsed < 1500);
EXCEPTION WHEN OTHERS THEN
  IF control_pid IS NOT NULL THEN
    BEGIN
      EXECUTE format(
        $q$COPY (SELECT %s) TO PROGRAM 'read pid; kill -CONT "$pid"'$q$,
        control_pid);
    EXCEPTION WHEN OTHERS THEN
      NULL;
    END;
  END IF;
  IF test_pid IS NOT NULL THEN
    BEGIN
      EXECUTE format(
        $q$COPY (SELECT %s) TO PROGRAM 'read pid; kill -CONT "$pid"'$q$,
        test_pid);
    EXCEPTION WHEN OTHERS THEN
      NULL;
    END;
  END IF;
  RAISE;
END
$test$;

SELECT control_result,
       round(control_elapsed_ms, 3) AS control_elapsed_ms,
       signaled_result,
       round(signaled_elapsed_ms, 3) AS signaled_elapsed_ms,
       reproduced
  FROM timeout_result;

\if :dblink_preexisting
\else
DROP EXTENSION dblink;
\endif
```

A representative recorded run produced:

```
CREATE EXTENSION
CREATE TABLE
DO
f|1001.698|f|179.005|t
DROP EXTENSION

WARNING:  backend with PID 85229 did not terminate within 1000 milliseconds
WARNING:  backend with PID 85232 did not terminate within 3000 milliseconds
```

The 1000 ms control lasted 1001.698 ms, whereas repeated SIGUSR1 wakeups
made the nominal 3000 ms wait return false after 179.005 ms. Another run
measured 1001.860 ms for the control and 171.359 ms for the signaled call.

The issue was reproduced on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on
Linux/amd64. pg_wait_until_termination() subtracts the requested 100 ms wait
interval after every WaitLatch return, even when the latch wakes before that
interval has elapsed. The termination signal is still sent; callers can work
around the premature synchronous return by polling for backend termination
or retrying without unrelated latch traffic. This occurs when
pg_terminate_backend waits with a positive timeout while unrelated signals
repeatedly wake the caller's latch.








view thread (2+ messages)  latest in thread

Message-ID: <19886-16f4759c592d0c00@postgresql.org>
Permalink:  ../19886-16f4759c592d0c00@postgresql.org/
Also on:    postgresql.org/message-id/19886-16f4759c592d0c00@postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: noreply@postgresql.org, imchifan8@gmail.com
  Subject: Re: BUG #19886: pg_terminate_backend timeout expires early after latch wakeups
  In-Reply-To: <19886-16f4759c592d0c00@postgresql.org>
  References: <19886-16f4759c592d0c00@postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox