public inbox for [email protected]
help / color / mirror / Atom feedFrom: Aya Iwata (Fujitsu) <[email protected]>
To: 'Peter Smith' <[email protected]>
To: Michael Paquier <[email protected]>
Cc: Chao Li <[email protected]>
Cc: Hayato Kuroda (Fujitsu) <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: RE: [PROPOSAL] Termination of Background Workers for ALTER/DROP DATABASE
Date: Tue, 21 Oct 2025 14:14:34 +0000
Message-ID: <OS7PR01MB11964AC66E36D8EAE1DAD1440EAF2A@OS7PR01MB11964.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <CAHut+PsN3jsKFtsnkwM+0x0Ak4g2dmCaQjb2r=GbX=T+AGRP2w@mail.gmail.com>
References: <[email protected]>
<OSCPR01MB1496614832F8014EC16FB78D2F5EEA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
<[email protected]>
<OSCPR01MB14966EC12277712131EB8EDF1F5EEA@OSCPR01MB14966.jpnprd01.prod.outlook.com>
<OS7PR01MB11964C8FE9CDCC0F4C9110988EAEEA@OS7PR01MB11964.jpnprd01.prod.outlook.com>
<[email protected]>
<OS7PR01MB11964C077A3E61F4887DD3D09EAEFA@OS7PR01MB11964.jpnprd01.prod.outlook.com>
<CAHut+PtbOP_80OPZXCUZO=-pBJSRTmHcQ2MnVTFov1meNbw18Q@mail.gmail.com>
<CAHut+Pt5BN0LDh7OzbNqh9+zqHBgsrLX+vh-gn+3FKYTFHMvhw@mail.gmail.com>
<TY3PR01MB11969CBD6DF3E0DB820AD4262EAE8A@TY3PR01MB11969.jpnprd01.prod.outlook.com>
<[email protected]>
<OS7PR01MB119642C7C0E94E767DB7000C1EAE9A@OS7PR01MB11964.jpnprd01.prod.outlook.com>
<CAHut+PsN3jsKFtsnkwM+0x0Ak4g2dmCaQjb2r=GbX=T+AGRP2w@mail.gmail.com>
Hi Peter-san, Michael-san,
Thank you for your comments.
I updated patch to v0009. Please review attached patch.
> -----Original Message-----
> From: Peter Smith <[email protected]>
> Sent: Monday, October 20, 2025 11:02 AM
> Some comments for the latest v8 patch.
>
> ======
> src/backend/postmaster/bgworker.c
>
> TerminateBgWorkersByBbOid:
>
> 1.
> +void
> +TerminateBgWorkersByDbOid(Oid oid)
>
> Now the function name is more explicit, but that is not a good reason
> to make the parameter name more vague.
>
> IMO the parameter should still be "dbOid" or "databaseId" instead of
> just "oid". (ditto for the extern in bgworker.h)
I agree with you. I reverted parameter name to databaseId.
> ======
> src/backend/storage/ipc/procarray.c
>
>
> CountOtherDBBackends:
>
> 2.
> + /*
> + * Usually, we try 50 times with 100ms sleep between tries, making 5 sec
> + * total wait. If requested, it would be reduced to 10 times to shorten the
> + * test time.
> + */
>
>
> The comment seemed vague to me. How about more like:
>
> /*
> * Retry up to 50 times with 100ms between attempts (max 5s total).
> * Can be reduced to 10 attempts (max 1s total) to speed up tests.
> */
Thank you. I updated this comment.
> 3.
> + for (tries = 0; tries < ntries; tries++)
>
> 'tries' can be declared as a for-loop variable.
I have declared "int" within the for-loop.
> ~~~
>
> 4.
> Something feels strange about this function name
> (CountOtherDBBackends) which suggests it is just for counting stuff,
> but in reality is more about exiting/terminating the workers. In fact
> retuns a boolean, not a count. Compare this with this similarly named
> "CountUserBackends" which really *is* doing what it says.
>
> Can we give this function a better name, or is that out of scope for this patch?
I think this is out of scope because existing code have terminated autovacuum process by SIGTERM.
It can be discussed separately.
I just added a comment to this function "background workers would also be terminated".
> ======
> src/test/modules/worker_spi/t/002_worker_terminate.pl
>
> 5.
> +# Firstly register an injection point to make the test faster. Normally, it
> +# spends more than 5 seconds because the backend retries, counting the
> number
> +# of connecting processes 50 times, but now the counting would be done only
> 10
> +# times. See CountOtherDBBackends().
> +$node->safe_psql('postgres', "CREATE EXTENSION injection_points;");
> +$node->safe_psql('postgres',
> + "SELECT injection_points_attach('reduce-ncounts', 'error');");
> +
>
> It seemed overkill to give details about what "normally" happens. I
> think it is enough to have a simple comment here:
>
> SUGGESTION
> The injection point 'reduce-ncounts' reduces the number of backend
> retries, allowing for shorter test runs. See CountOtherDBBackends().
Thank you for your suggestion. I updated this comment.
> -----Original Message-----
> From: Michael Paquier <[email protected]>
> Sent: Monday, October 20, 2025 1:33 PM
> On Mon, Oct 20, 2025 at 01:01:31PM +1100, Peter Smith wrote:
> > Some comments for the latest v8 patch.
>
> The comments of Peter apply to comments and parameters. I am not
> going down to these details in this message, these can be infinitely
> tuned.
>
> The injection point integration looks correct. You are checking the
> compile flag and if the extension is available in the installation
> path, which should be enough.
>
> + if (IS_INJECTION_POINT_ATTACHED("reduce-ncounts"))
> + ntries = 10;
>
> 1s is much faster than the default of 5s, still I am wondering if this
> cannot be brought down a bit more. Dropping the worker still around
> after the first test with CREATE DATABASE works here.
Thank you. I updated ntries to 3.
> +# Confirm a background worker is still running
> +$node->safe_psql(
> + "postgres", qq(
> + SELECT count(1) FROM pg_stat_activity
> + WHERE backend_type = 'worker_spi dynamic';));
>
> This does not check that the worker that does not have the flag set is
> still running: you are not feeding the output of this query to an is()
> test.
>
> + is($result, 't', "dynamic bgworker launched");
>
> In launch_bgworker(), this uses the same test description for all the
> callers of this subroutine. Let's prefix it with $testcase.
I added $testcase. Is it same as your expectations?
> +void
> +TerminateBgWorkersByDbOid(Oid oid)
>
> FWIW, while reading this code, I was wondering about one improvement
> that could show benefits for more extension code than only what we are
> discussing here because external code has no access to
> BackgroundWorkerSlot while holding the LWLock BackgroundWorkerLock in
> a single loop, by rewriting this new routine with something like that:
> void TerminateBackgroundWorkerMatchin(
> bool (*do_terminate) (int pid, BackgroundWorker *, Datum))
>
> Then the per-database termination would be a custom routine, defined
> also in bgworker.c. Other extension code could define their own
> filtering callback routine. Just an idea in passing, to let extension
> code take more actions on bgworker slots in use-based on a PGPROC
> entry, like a role ID for example, or it could be a different factor.
> Feel free to dislike such a funky idea if you do not like it and say
> so, of course.
Thank you for your advice.
I'd like to address that, but I couldn't figure out how to do it on my own.
Could you please describe it more?
Regards,
Aya Iwata
Fujitsu Limited
Attachments:
[application/octet-stream] v0009-0001-Allow-background-workers-to-be-terminated.patch (13.8K, 2-v0009-0001-Allow-background-workers-to-be-terminated.patch)
download | inline diff:
From 4616732872339c0a39f0647b9455617f4d3ab719 Mon Sep 17 00:00:00 2001
From: "iwata.aya" <[email protected]>
Date: Thu, 11 Sep 2025 21:16:51 +0900
Subject: [PATCH v0009] Allow background workers to be terminated at DROP
DATABASE
---
doc/src/sgml/bgworker.sgml | 19 +++
src/backend/postmaster/bgworker.c | 40 +++++
src/backend/storage/ipc/procarray.c | 28 +++-
src/include/postmaster/bgworker.h | 8 +
src/test/modules/worker_spi/Makefile | 4 +
src/test/modules/worker_spi/meson.build | 4 +
.../worker_spi/t/002_worker_terminate.pl | 140 ++++++++++++++++++
.../modules/worker_spi/worker_spi--1.0.sql | 3 +-
src/test/modules/worker_spi/worker_spi.c | 5 +
9 files changed, 245 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/worker_spi/t/002_worker_terminate.pl
diff --git a/doc/src/sgml/bgworker.sgml b/doc/src/sgml/bgworker.sgml
index 2c393385a91..6f4fc57e3d9 100644
--- a/doc/src/sgml/bgworker.sgml
+++ b/doc/src/sgml/bgworker.sgml
@@ -108,6 +108,25 @@ typedef struct BackgroundWorker
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>BGWORKER_EXIT_AT_DATABASE_CHANGE</literal></term>
+ <listitem>
+ <para>
+ <indexterm><primary>BGWORKER_EXIT_AT_DATABASE_CHANGE</primary></indexterm>
+ Requests termination of the background worker when its connected database is
+ dropped, renamed, moved to a different tablespace, or used as a template for
+ <command>CREATE DATABASE</command>. Specifically, the postmaster sends a
+ termination signal when any of these commands affect the worker's database:
+ <command>DROP DATABASE</command>,
+ <command>ALTER DATABASE RENAME TO</command>,
+ <command>ALTER DATABASE SET TABLESPACE</command>, or
+ <command>CREATE DATABASE</command>.
+ Requires both <literal>BGWORKER_SHMEM_ACCESS</literal> and
+ <literal>BGWORKER_BACKEND_DATABASE_CONNECTION</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</para>
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 1ad65c237c3..462ac749c76 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -26,6 +26,7 @@
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
+#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "storage/shmem.h"
#include "tcop/tcopprot.h"
@@ -1396,3 +1397,42 @@ GetBackgroundWorkerTypeByPid(pid_t pid)
return result;
}
+
+/*
+ * Terminate all background workers connected to the given database, if they
+ * had requested it.
+ */
+void
+TerminateBgWorkersByDbOid(Oid databaseId)
+{
+ bool signal_postmaster = false;
+
+ LWLockAcquire(BackgroundWorkerLock, LW_EXCLUSIVE);
+
+ /*
+ * Iterate through slots, looking for workers connected to the given
+ * database.
+ */
+ for (int slotno = 0; slotno < BackgroundWorkerData->total_slots; ++slotno)
+ {
+ BackgroundWorkerSlot *slot = &BackgroundWorkerData->slot[slotno];
+
+ if (slot->in_use &&
+ (slot->worker.bgw_flags & BGWORKER_EXIT_AT_DATABASE_CHANGE))
+ {
+ PGPROC *proc = BackendPidGetProc(slot->pid);
+
+ if (proc && proc->databaseId == databaseId)
+ {
+ slot->terminate = true;
+ signal_postmaster = true;
+ }
+ }
+ }
+
+ LWLockRelease(BackgroundWorkerLock);
+
+ /* Make sure the postmaster notices the change to shared memory. */
+ if (signal_postmaster)
+ SendPostmasterSignal(PMSIGNAL_BACKGROUND_WORKER_CHANGE);
+}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 200f72c6e25..0044f8cbdf6 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -56,11 +56,13 @@
#include "catalog/pg_authid.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgworker.h"
#include "port/pg_lfind.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
+#include "utils/injection_point.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
@@ -3689,8 +3691,9 @@ CountUserBackends(Oid roleid)
* CountOtherDBBackends -- check for other backends running in the given DB
*
* If there are other backends in the DB, we will wait a maximum of 5 seconds
- * for them to exit. Autovacuum backends are encouraged to exit early by
- * sending them SIGTERM, but normal user backends are just waited for.
+ * for them to exit. Autovacuum backends and background workers are encouraged
+ * to exit early by sending them SIGTERM, but normal user backends are just
+ * waited for.
*
* The current backend is always ignored; it is caller's responsibility to
* check whether the current backend uses the given DB, if it's important.
@@ -3715,10 +3718,19 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
#define MAXAUTOVACPIDS 10 /* max autovacs to SIGTERM per iteration */
int autovac_pids[MAXAUTOVACPIDS];
- int tries;
- /* 50 tries with 100ms sleep between tries makes 5 sec total wait */
- for (tries = 0; tries < 50; tries++)
+ /*
+ * Retry up to 50 times with 100ms between attempts (max 5s total). Can be
+ * reduced to 3 attempts (max 0.3s total) to speed up tests.
+ */
+ int ntries = 50;
+
+#ifdef USE_INJECTION_POINTS
+ if (IS_INJECTION_POINT_ATTACHED("reduce-ncounts"))
+ ntries = 3;
+#endif
+
+ for (int tries = 0; tries < ntries; tries++)
{
int nautovacs = 0;
bool found = false;
@@ -3768,6 +3780,12 @@ CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared)
for (index = 0; index < nautovacs; index++)
(void) kill(autovac_pids[index], SIGTERM); /* ignore any error */
+ /*
+ * Terminate all background workers for this database, if they had
+ * requested it (BGWORKER_EXIT_AT_DATABASE_CHANGE)
+ */
+ TerminateBgWorkersByDbOid(databaseId);
+
/* sleep, then try again */
pg_usleep(100 * 1000L); /* 100ms */
}
diff --git a/src/include/postmaster/bgworker.h b/src/include/postmaster/bgworker.h
index 058667a47a0..8c69df432a5 100644
--- a/src/include/postmaster/bgworker.h
+++ b/src/include/postmaster/bgworker.h
@@ -59,6 +59,13 @@
*/
#define BGWORKER_BACKEND_DATABASE_CONNECTION 0x0002
+/*
+ * Exit the bgworker if its database is involved in a CREATE, ALTER or DROP
+ * database command.
+ * Requires BGWORKER_SHMEM_ACCESS and BGWORKER_BACKEND_DATABASE_CONNECTION.
+ */
+#define BGWORKER_EXIT_AT_DATABASE_CHANGE 0x0004
+
/*
* This class is used internally for parallel queries, to keep track of the
* number of active parallel workers and make sure we never launch more than
@@ -128,6 +135,7 @@ extern const char *GetBackgroundWorkerTypeByPid(pid_t pid);
/* Terminate a bgworker */
extern void TerminateBackgroundWorker(BackgroundWorkerHandle *handle);
+extern void TerminateBgWorkersByDbOid(Oid databaseId);
/* This is valid in a running worker */
extern PGDLLIMPORT BackgroundWorker *MyBgworkerEntry;
diff --git a/src/test/modules/worker_spi/Makefile b/src/test/modules/worker_spi/Makefile
index 024b34cdbb3..e7c5c059e32 100644
--- a/src/test/modules/worker_spi/Makefile
+++ b/src/test/modules/worker_spi/Makefile
@@ -6,6 +6,10 @@ EXTENSION = worker_spi
DATA = worker_spi--1.0.sql
PGFILEDESC = "worker_spi - background worker example"
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/src/test/modules/worker_spi/meson.build b/src/test/modules/worker_spi/meson.build
index d673ece48a0..5ba66051396 100644
--- a/src/test/modules/worker_spi/meson.build
+++ b/src/test/modules/worker_spi/meson.build
@@ -26,8 +26,12 @@ tests += {
'sd': meson.current_source_dir(),
'bd': meson.current_build_dir(),
'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
'tests': [
't/001_worker_spi.pl',
+ 't/002_worker_terminate.pl'
],
},
}
diff --git a/src/test/modules/worker_spi/t/002_worker_terminate.pl b/src/test/modules/worker_spi/t/002_worker_terminate.pl
new file mode 100644
index 00000000000..9671f7e5770
--- /dev/null
+++ b/src/test/modules/worker_spi/t/002_worker_terminate.pl
@@ -0,0 +1,140 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test background workers can be terminated by db commands
+
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# This test depends on injection points to detect whether background workers
+# remain.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Ensure the worker_spi dynamic worker is launched on the specified database
+sub launch_bgworker
+{
+ my ($node, $database, $testcase, $request_terminate) = @_;
+ my $offset = -s $node->logfile;
+
+ # Launch a background worker on the given database
+ my $result = $node->safe_psql(
+ $database, qq(
+ SELECT worker_spi_launch($testcase, oid, 0, '{}', $request_terminate) IS NOT NULL
+ FROM pg_database WHERE datname = '$database';
+ ));
+ is($result, 't', "dynamic bgworker $testcase launched");
+
+ # Check the worker is surely initialized
+ $node->wait_for_log(
+ qr/LOG: worker_spi dynamic worker $testcase initialized with .*\..*/,
+ $offset);
+}
+
+# Run the given query and verify the background worker can be terminated
+sub run_db_command
+{
+ my ($node, $command, $testname) = @_;
+ my $offset = -s $node->logfile;
+
+ $node->safe_psql('postgres', $command);
+
+ $node->wait_for_log(
+ qr/terminating background worker \"worker_spi dynamic\" due to administrator command/,
+ $offset);
+
+ note("background worker can be terminated at $testname");
+}
+
+my $node = PostgreSQL::Test::Cluster->new('mynode');
+$node->init;
+$node->start;
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+$node->safe_psql('postgres', 'CREATE EXTENSION worker_spi;');
+
+# Launch a background worker without BGWORKER_EXIT_AT_DATABASE_CHANGE
+launch_bgworker($node, 'postgres', 0, "false");
+
+# Ensure CREATE DATABASE WITH TEMPLATE fails because background worker retains
+
+# The injection point 'reduce-ncounts' reduces the number of backend
+# retries, allowing for shorter test runs. See CountOtherDBBackends().
+$node->safe_psql('postgres', "CREATE EXTENSION injection_points;");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('reduce-ncounts', 'error');");
+
+my $stderr;
+
+$node->psql(
+ 'postgres',
+ "CREATE DATABASE testdb WITH TEMPLATE postgres",
+ stderr => \$stderr);
+ok( $stderr =~
+ "source database \"postgres\" is being accessed by other users",
+ "background worker blocked the database creation");
+
+# Confirm a background worker is still running
+my $result = $node->safe_psql(
+ "postgres", qq(
+ SELECT count(1) FROM pg_stat_activity
+ WHERE backend_type = 'worker_spi dynamic';));
+
+is($result, '1',
+ "background worker is still running after CREATE DATABASE WITH TEMPLATE");
+
+# Terminate the worker for upcoming tests
+$node->safe_psql(
+ "postgres", qq(
+ SELECT pg_terminate_backend(pid)
+ FROM pg_stat_activity WHERE backend_type = 'worker_spi dynamic';));
+
+# The injection point won't be used anymore, release it.
+$node->safe_psql('postgres',
+ "SELECT injection_points_detach('reduce-ncounts');");
+
+# Ensure BGWORKER_EXIT_AT_DATABASE_CHANGE allows background workers to be
+# terminated at some database manipulations.
+#
+# Testcase 1: CREATE DATABASE WITH TEMPLATE
+launch_bgworker($node, 'postgres', 1, "true");
+run_db_command(
+ $node,
+ "CREATE DATABASE testdb WITH TEMPLATE postgres",
+ "CREATE DATABASE WITH TEMPLATE");
+
+# Testcase 2: ALTER DATABASE RENAME
+launch_bgworker($node, 'testdb', 2, "true");
+run_db_command(
+ $node,
+ "ALTER DATABASE testdb RENAME TO renameddb",
+ "ALTER DATABASE RENAME");
+
+# Preparation for the next test; create another tablespace
+my $tablespace = PostgreSQL::Test::Utils::tempdir;
+$node->safe_psql('postgres',
+ "CREATE TABLESPACE test_tablespace LOCATION '$tablespace'");
+
+# Testcase 3: ALTER DATABASE SET TABLESPACE
+launch_bgworker($node, 'renameddb', 3, "true");
+run_db_command(
+ $node,
+ "ALTER DATABASE renameddb SET TABLESPACE test_tablespace",
+ "ALTER DATABASE SET TABLESPACE");
+
+# Testcase 4: DROP DATABASE
+launch_bgworker($node, 'renameddb', 4, "true");
+run_db_command($node, "DROP DATABASE renameddb", "DROP DATABASE");
+
+done_testing();
diff --git a/src/test/modules/worker_spi/worker_spi--1.0.sql b/src/test/modules/worker_spi/worker_spi--1.0.sql
index 84deb6199f6..3d12de37bea 100644
--- a/src/test/modules/worker_spi/worker_spi--1.0.sql
+++ b/src/test/modules/worker_spi/worker_spi--1.0.sql
@@ -7,7 +7,8 @@
CREATE FUNCTION worker_spi_launch(index int4,
dboid oid DEFAULT 0,
roleoid oid DEFAULT 0,
- flags text[] DEFAULT '{}')
+ flags text[] DEFAULT '{}',
+ request_termination boolean DEFAULT false)
RETURNS pg_catalog.int4 STRICT
AS 'MODULE_PATHNAME'
LANGUAGE C;
diff --git a/src/test/modules/worker_spi/worker_spi.c b/src/test/modules/worker_spi/worker_spi.c
index bea8339f464..2912abe6cce 100644
--- a/src/test/modules/worker_spi/worker_spi.c
+++ b/src/test/modules/worker_spi/worker_spi.c
@@ -404,10 +404,15 @@ worker_spi_launch(PG_FUNCTION_ARGS)
Size ndim;
int nelems;
Datum *datum_flags;
+ bool request_termination = PG_GETARG_BOOL(4);
memset(&worker, 0, sizeof(worker));
worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
+
+ if (request_termination)
+ worker.bgw_flags |= BGWORKER_EXIT_AT_DATABASE_CHANGE;
+
worker.bgw_start_time = BgWorkerStart_RecoveryFinished;
worker.bgw_restart_time = BGW_NEVER_RESTART;
sprintf(worker.bgw_library_name, "worker_spi");
--
2.39.3
view thread (67+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: RE: [PROPOSAL] Termination of Background Workers for ALTER/DROP DATABASE
In-Reply-To: <OS7PR01MB11964AC66E36D8EAE1DAD1440EAF2A@OS7PR01MB11964.jpnprd01.prod.outlook.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox