pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedproblem with segmentation fault error
11+ messages / 5 participants
[nested] [flat]
* problem with segmentation fault error
@ 2010-01-27 21:27 pg2@onet.eu
2010-01-27 23:59 ` Re: problem with segmentation fault error Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 11+ messages in thread
From: pg2@onet.eu @ 2010-01-27 21:27 UTC (permalink / raw)
To: pgsql-bugs
Hello,
I've got problem with PostgreSQL crash (today it was twice).
Usually, average load5 is about 16-20. Today I've run new application and load has increased dramatically.
First time before crash load was fast rising from 30 to 70 (log below)
2010-01-27 10:34:01 CET [3931]: [9-1] host=,user=,db= LOG: server process (PID 24454) was terminated by signal 11: Segmentation fault
2010-01-27 10:34:01 CET [3931]: [10-1] host=,user=,db= LOG: terminating any other active server processes
2010-01-27 10:34:01 CET [21906]: [1-1] host=X.X.X.X,user=pgmed,db=pgmed WARNING: terminating connection because of crash of another server process
2010-01-27 10:34:01 CET [21906]: [2-1] host=X.X.X.X,user=pgmed,db=pgmed DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Second time, load was about 70-80 and processor utilization was almost 100% from 1 hour (log below)
2010-01-27 18:46:06 CET [3931]: [16-1] host=,user=,db= LOG: server process (PID 28357) was terminated by signal 11: Segmentation fault
2010-01-27 18:46:06 CET [3931]: [17-1] host=,user=,db= LOG: terminating any other active server processes
2010-01-27 18:46:06 CET [26377]: [3-1] host=X.X.X.X,user=pgmed_app,db=pgmed WARNING: terminating connection because of crash of another server process
2010-01-27 18:46:06 CET [26377]: [4-1] host=X.X.X.X,user=pgmed_app,db=pgmed DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
I have 8.4.0 version.
Do you have any idea where problem could be? Is it hardware or rather PostgreSQL configuration problem?
I'll be grateful for any help.
Best regards
PG
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: problem with segmentation fault error
2010-01-27 21:27 problem with segmentation fault error pg2@onet.eu
@ 2010-01-27 23:59 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 11+ messages in thread
From: Tom Lane @ 2010-01-27 23:59 UTC (permalink / raw)
To: pg2@onet.eu; +Cc: pgsql-bugs
pg2@onet.eu writes:
> 2010-01-27 10:34:01 CET [3931]: [9-1] host=,user=,db= LOG: server process (PID 24454) was terminated by signal 11: Segmentation fault
> I have 8.4.0 version.
The *first* thing to do is update to 8.4.2 to see if the problem was
already fixed.
If not, you need to try to gather some data. What query was running in
the crashed backend would be useful to know, and a stack trace would be
even better.
> Do you have any idea where problem could be? Is it hardware or rather PostgreSQL configuration problem?
It could be a hardware problem or a software bug. How reproducible is
it?
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* cache lookup failed dropping public schema with trgm index
@ 2023-08-21 18:40 Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:09 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 11+ messages in thread
From: Wyatt Alt @ 2023-08-21 18:40 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Hi,
This reproduces on 15.4 and 13.12:
create table foo(t text);
create extension pg_trgm;
create index on foo using gist(t gist_trgm_ops);
drop schema public cascade;
NOTICE: drop cascades to 2 other objects
DETAIL: drop cascades to table foo
drop cascades to extension pg_trgm
ERROR: cache lookup failed for function 1195999
Time: 20.968 ms
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
@ 2023-08-21 22:09 ` Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 11+ messages in thread
From: Michael Paquier @ 2023-08-21 22:09 UTC (permalink / raw)
To: Wyatt Alt <wyatt.alt@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
On Mon, Aug 21, 2023 at 11:40:15AM -0700, Wyatt Alt wrote:
> This reproduces on 15.4 and 13.12:
>
> create table foo(t text);
> create extension pg_trgm;
> create index on foo using gist(t gist_trgm_ops);
> drop schema public cascade;
Indeed, reproduced here down to 13. 12 and older versions don't react
like that. I'll go bisect that..
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
@ 2023-08-21 22:36 ` Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 11+ messages in thread
From: Andres Freund @ 2023-08-21 22:36 UTC (permalink / raw)
To: Wyatt Alt <wyatt.alt@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
Hi,
On 2023-08-21 11:40:15 -0700, Wyatt Alt wrote:
> This reproduces on 15.4 and 13.12:
Also reproduces on HEAD.
> create table foo(t text);
> create extension pg_trgm;
> create index on foo using gist(t gist_trgm_ops);
> drop schema public cascade;
>
> NOTICE: drop cascades to 2 other objects
> DETAIL: drop cascades to table foo
> drop cascades to extension pg_trgm
> ERROR: cache lookup failed for function 1195999
> Time: 20.968 ms
It also seems to work without even involving a drop schema. Just dropping
pg_trgm with cascade is sufficient.
<several wrong theories>
I think we might primarily dealing with missing invalidations. Dropping
objects is scheduled in an order where we first drop
schedule deletion of function 10 (text, text) of operator family t.gist_trgm_ops for access method gist: t.gtrgm_options(internal) at 0
schedule deletion of function t.gtrgm_options(internal) at 1
and then later
schedule deletion of index t.foo_t at 32
During the index deletion we try to initialize the access method. But haven't
performed sufficient invalidation and still think "function 10"
exists. Calling it then causes the error.
One can "verify" this theory by adding an InvalidateSystemCaches() at the end
of deleteOneObject(). That "fixes" the issue. This also explains why dropping
pg_trgm in a new session works - we don't have old cache entries that could be
out of date.
Not quite sure where we are dropping the ball with invalidations yet.
However, I suspect there's more wrong than this, albeit perhaps not
problematic in a huge way. While debugging I added a getObjectDescription()
description call to deleteObjectsInList() *before* calling
deleteOneObject(). That fails when dropping pg_trgmp, because we end up
dropping the type gtrgm before trgm_out(). The reason this happens is because
we reach gtrgm_out() via the extension dependency, rather than via the
type. When recursing to gtrgm_out(), we recurse to type gtrgm, recurse to
gtrgm_in(), schedule it for deletion, then recurse to gtrgm_out(), but find
it's in the stack and do *not* schedule it for deletion, before scheduling
gtrgm for deletion. Only then we delete gtrgm_out().
Now, this isn't a real issue in practice (without such a debugging statement,
which likely can't work in some cases), but I strongly suspect that it
indicates a scheduling order issue that's more widespread. Despite, I think,
correct dependencies, we end up with a topologically inconsistent drop
order. There aren't any cycles in the directed dependency graph from what I
can see.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
@ 2023-08-21 22:55 ` Michael Paquier <michael@paquier.xyz>
2023-08-21 23:13 ` Re: cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 23:27 ` Re: cache lookup failed dropping public schema with trgm index Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 11+ messages in thread
From: Michael Paquier @ 2023-08-21 22:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Wyatt Alt <wyatt.alt@gmail.com>; pgsql-bugs@lists.postgresql.org
On Mon, Aug 21, 2023 at 03:36:10PM -0700, Andres Freund wrote:
> It also seems to work without even involving a drop schema. Just dropping
> pg_trgm with cascade is sufficient.
FWIW, after a bisect I can see that 911e7020 is the origin of the
failure (`git bisect start b5d69b7 9e1c9f9` based on two merge-bases).
> Now, this isn't a real issue in practice (without such a debugging statement,
> which likely can't work in some cases), but I strongly suspect that it
> indicates a scheduling order issue that's more widespread. Despite, I think,
> correct dependencies, we end up with a topologically inconsistent drop
> order. There aren't any cycles in the directed dependency graph from what I
> can see.
Yeah, guess so. I was first betting on a missing shared inval here.
Now note that for example, this command works:
psql -v ON_ERROR_STOP=1 -c 'create table foo(t text); create extension pg_trgm; create index on foo using gist(t gist_trgm_ops);create index on foo using gist(t gist_trgm_ops); drop schema public cascade;'
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
@ 2023-08-21 23:13 ` Wyatt Alt <wyatt.alt@gmail.com>
1 sibling, 0 replies; 11+ messages in thread
From: Wyatt Alt @ 2023-08-21 23:13 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-bugs@lists.postgresql.org
Thanks for having a look. Note also that switching the first two lines of
my example prevents the failure.
On Mon, Aug 21, 2023 at 3:55 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Aug 21, 2023 at 03:36:10PM -0700, Andres Freund wrote:
> > It also seems to work without even involving a drop schema. Just dropping
> > pg_trgm with cascade is sufficient.
>
> FWIW, after a bisect I can see that 911e7020 is the origin of the
> failure (`git bisect start b5d69b7 9e1c9f9` based on two merge-bases).
>
> > Now, this isn't a real issue in practice (without such a debugging
> statement,
> > which likely can't work in some cases), but I strongly suspect that it
> > indicates a scheduling order issue that's more widespread. Despite, I
> think,
> > correct dependencies, we end up with a topologically inconsistent drop
> > order. There aren't any cycles in the directed dependency graph from
> what I
> > can see.
>
> Yeah, guess so. I was first betting on a missing shared inval here.
> Now note that for example, this command works:
> psql -v ON_ERROR_STOP=1 -c 'create table foo(t text); create extension
> pg_trgm; create index on foo using gist(t gist_trgm_ops);create index on
> foo using gist(t gist_trgm_ops); drop schema public cascade;'
> --
> Michael
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
@ 2023-08-21 23:27 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-22 01:43 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-22 18:45 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 11+ messages in thread
From: Tom Lane @ 2023-08-21 23:27 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Wyatt Alt <wyatt.alt@gmail.com>; pgsql-bugs@lists.postgresql.org
Michael Paquier <michael@paquier.xyz> writes:
> On Mon, Aug 21, 2023 at 03:36:10PM -0700, Andres Freund wrote:
>> It also seems to work without even involving a drop schema. Just dropping
>> pg_trgm with cascade is sufficient.
> FWIW, after a bisect I can see that 911e7020 is the origin of the
> failure (`git bisect start b5d69b7 9e1c9f9` based on two merge-bases).
Hmm. I see that 911e7020 modified pg_trgm's install script with
+ALTER OPERATOR FAMILY gist_trgm_ops USING gist
+ADD FUNCTION 10 (text) gtrgm_options (internal);
I wonder whether that correctly adds a dependency to ensure the
opfamily is dropped before the function.
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 23:27 ` Re: cache lookup failed dropping public schema with trgm index Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-08-22 01:43 ` Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 11+ messages in thread
From: Michael Paquier @ 2023-08-22 01:43 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Wyatt Alt <wyatt.alt@gmail.com>; pgsql-bugs@lists.postgresql.org
On Mon, Aug 21, 2023 at 07:27:19PM -0400, Tom Lane wrote:
> Hmm. I see that 911e7020 modified pg_trgm's install script with
>
> +ALTER OPERATOR FAMILY gist_trgm_ops USING gist
> +ADD FUNCTION 10 (text) gtrgm_options (internal);
>
> I wonder whether that correctly adds a dependency to ensure the
> opfamily is dropped before the function.
The dependencies look OK seen from here when it comes to the
extension, the indexes or the option function. See, for instance:
=# SELECT pg_describe_object(classid, objid, objsubid) as obj,
pg_describe_object(refclassid,refobjid,refobjsubid) as objref,
deptype FROM pg_depend
WHERE (classid = 'pg_proc'::regclass AND objid = 'gtrgm_options'::regproc)
OR (refclassid = 'pg_proc'::regclass AND refobjid = 'gtrgm_options'::regproc);
-[ RECORD 1]------------------------------------------------------------------------------------------------------
obj | function gtrgm_options(internal)
objref | schema public
deptype | n
-[ RECORD 2
]------------------------------------------------------------------------------------------------------
obj | function gtrgm_options(internal)
objref | extension pg_trgm
deptype | e
-[ RECORD 3]------------------------------------------------------------------------------------------------------
obj | function 10 (text, text) of operator family gist_trgm_ops for access method gist: gtrgm_options(internal)
objref | function gtrgm_options(internal)
deptype | a
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 23:27 ` Re: cache lookup failed dropping public schema with trgm index Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-08-22 18:45 ` Andres Freund <andres@anarazel.de>
2023-08-28 02:15 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 11+ messages in thread
From: Andres Freund @ 2023-08-22 18:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; Wyatt Alt <wyatt.alt@gmail.com>; pgsql-bugs@lists.postgresql.org
Hi,
On 2023-08-21 19:27:19 -0400, Tom Lane wrote:
> Michael Paquier <michael@paquier.xyz> writes:
> > On Mon, Aug 21, 2023 at 03:36:10PM -0700, Andres Freund wrote:
> >> It also seems to work without even involving a drop schema. Just dropping
> >> pg_trgm with cascade is sufficient.
>
> > FWIW, after a bisect I can see that 911e7020 is the origin of the
> > failure (`git bisect start b5d69b7 9e1c9f9` based on two merge-bases).
>
> Hmm. I see that 911e7020 modified pg_trgm's install script with
>
> +ALTER OPERATOR FAMILY gist_trgm_ops USING gist
> +ADD FUNCTION 10 (text) gtrgm_options (internal);
>
> I wonder whether that correctly adds a dependency to ensure the
> opfamily is dropped before the function.
It's not quite that, I think. Whether it fails or succeeds depends on the
state of the system caches. The dependencies lead to the equivalent of
ALTER OPERATOR FAMILY ... USING ... DROP
being performed, before dropping the index.
I think there are at least three levels of problems here:
- I don't think we currently properly force index relcache entries to be
invalidated when the relevant opfamily changes
- LookupOpclassInfo() doesn't have *any* invalidation support, so even if we
were to perform invalidation, we'd still return a potentially stale
entry. The function says:
* Note there is no provision for flushing the cache. This is OK at the
* moment because there is no way to ALTER any interesting properties of an
* existing opclass --- all you can do is drop it, which will result in
* a useless but harmless dead entry in the cache.
But that's not true (amymore?), because it does do pg_amproc lookups, and
they *can* change.
- Minor: Even if you force LookupOpclassInfo() to perform lookups again, it
doesn't work, because when an amproc entry doesn't exist anymore, the old
value in opcentry->supportProcs[i] survives, because nothing resets it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: cache lookup failed dropping public schema with trgm index
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 23:27 ` Re: cache lookup failed dropping public schema with trgm index Tom Lane <tgl@sss.pgh.pa.us>
2023-08-22 18:45 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
@ 2023-08-28 02:15 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 11+ messages in thread
From: Michael Paquier @ 2023-08-28 02:15 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Wyatt Alt <wyatt.alt@gmail.com>; pgsql-bugs@lists.postgresql.org
On Tue, Aug 22, 2023 at 11:45:58AM -0700, Andres Freund wrote:
> It's not quite that, I think. Whether it fails or succeeds depends on the
> state of the system caches. The dependencies lead to the equivalent of
> ALTER OPERATOR FAMILY ... USING ... DROP
> being performed, before dropping the index.
>
> I think there are at least three levels of problems here:
>
> - I don't think we currently properly force index relcache entries to be
> invalidated when the relevant opfamily changes
>
> - LookupOpclassInfo() doesn't have *any* invalidation support, so even if we
> were to perform invalidation, we'd still return a potentially stale
> entry. The function says:
>
> * Note there is no provision for flushing the cache. This is OK at the
> * moment because there is no way to ALTER any interesting properties of an
> * existing opclass --- all you can do is drop it, which will result in
> * a useless but harmless dead entry in the cache.
>
> But that's not true (amymore?), because it does do pg_amproc lookups, and
> they *can* change.
>
> - Minor: Even if you force LookupOpclassInfo() to perform lookups again, it
> doesn't work, because when an amproc entry doesn't exist anymore, the old
> value in opcentry->supportProcs[i] survives, because nothing resets it.
Another thing: we also need to think harder about
RelationReloadIndexInfo() which is the cheap path for cache reload for
the index information. This is quite different from rd_options,
though, because we are going to need an extra facility to look at the
pg_attribute tuples for the indexes to get access to the new option
options? We touch that a bit in RelationBuildTupleDesc(), but not in
this cheap reload path. This is going to require more infrastructure,
at quick glance..
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2023-08-28 02:15 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-01-27 21:27 problem with segmentation fault error pg2@onet.eu
2010-01-27 23:59 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-21 18:40 cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 22:09 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 22:36 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-21 22:55 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-21 23:13 ` Re: cache lookup failed dropping public schema with trgm index Wyatt Alt <wyatt.alt@gmail.com>
2023-08-21 23:27 ` Re: cache lookup failed dropping public schema with trgm index Tom Lane <tgl@sss.pgh.pa.us>
2023-08-22 01:43 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
2023-08-22 18:45 ` Re: cache lookup failed dropping public schema with trgm index Andres Freund <andres@anarazel.de>
2023-08-28 02:15 ` Re: cache lookup failed dropping public schema with trgm index Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox