public inbox for [email protected]
help / color / mirror / Atom feedadd_partial_path() may remove dominated path but still in use
51+ messages / 10 participants
[nested] [flat]
* add_partial_path() may remove dominated path but still in use
@ 2018-12-28 04:21 Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2018-12-28 04:21 UTC (permalink / raw)
To: pgsql-hackers
Hello,
I've investigated a crash report of PG-Strom for a few days, then I doubt
add_partial_path() can unexpectedly release dominated old partial path
but still referenced by other Gather node, and it leads unexpected system
crash.
Please check at the gpuscan.c:373
https://github.com/heterodb/pg-strom/blob/master/src/gpuscan.c#L373
The create_gpuscan_path() constructs a partial custom-path, then it is
added to the partial_pathlist of the baserel.
If both of old and new partial paths have no pathkeys and old-path has
larger cost, add_partial_path detaches the old path from the list, then
calls pfree() to release old_path itself.
On the other hands, the baserel may have GatherPath which references
the partial-path on its pathlist. Here is no check whether the old partial-
paths are referenced by others, or not.
To ensure my assumption, I tried to inject elog() before/after the call of
add_partial_path() and just before the pfree(old_path) in add_partial_path().
----------------------------------------------------------------
dbt3=# explain select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < cast(date '1994-01-01' + interval '1 year' as date)
and l_discount between 0.08 - 0.01 and 0.08 + 0.01
and l_quantity < 24 limit 1;
INFO: GpuScan:389 GATHER(0x28f3c30), SUBPATH(0x28f3f88): {GATHERPATH
:pathtype 44 :parent_relids (b 1) :required_outer (b) :parallel_aware
false :parallel_safe false :parallel_workers 0 :rows 151810
:startup_cost 1000.00 :total_cost 341760.73 :pathkeys <> :subpath
{PATH :pathtype 18 :parent_relids (b 1) :required_outer (b)
:parallel_aware true :parallel_safe true :parallel_workers 2 :rows
63254 :startup_cost 0.00 :total_cost 325579.73 :pathkeys <>}
:single_copy false :num_workers 2}
INFO: add_partial_path:830 old_path(0x28f3f88) is removed
WARNING: could not dump unrecognized node type: 2139062143
INFO: GpuScan:401 GATHER(0x28f3c30), SUBPATH(0x28f3f88): {GATHERPATH
:pathtype 44 :parent_relids (b 1) :required_outer (b) :parallel_aware
false :parallel_safe false :parallel_workers 0 :rows 151810
:startup_cost 1000.00 :total_cost 341760.73 :pathkeys <> :subpath
{(HOGE)} :single_copy false :num_workers 2}
----------------------------------------------------------------
At the L389, GatherPath in the baresel->pathlist is healthy. Its
subpath (0x28f3f88) is
a valid T_Scan path node.
Then, gpuscan.c adds a cheaper path-node so add_partial_path()
considers the above
subpath (0x28f3f88) is dominated by the new custom-path, and release it.
So, elog() at L401 says subpath has unrecognized node type: 2139062143
== 0x7f7f7f7f
that implies the memory region was already released by pfree().
Reference counter or other mechanism to tack referenced paths may be an idea
to avoid unintentional release of path-node.
On the other hands, it seems to me the pfree() at add_path /
add_partial_path is not
a serious memory management because other objects referenced by the path-node
are not released here.
It is sufficient if we detach dominated path-node from the pathlist /
partial_pathlist.
How about your opinions?
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2018-12-28 16:44 ` Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Tom Lane @ 2018-12-28 16:44 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: pgsql-hackers
Kohei KaiGai <[email protected]> writes:
> I've investigated a crash report of PG-Strom for a few days, then I doubt
> add_partial_path() can unexpectedly release dominated old partial path
> but still referenced by other Gather node, and it leads unexpected system
> crash.
Hm. This seems comparable to the special case in plain add_path, where it
doesn't attempt to free IndexPaths because of the risk that they're still
referenced. So maybe we should just drop the pfree here.
However, first I'd like to know why this situation is arising in the first
place. To have the situation you're describing, we'd have to have
attempted to make some Gather paths before we have all the partial paths
for the relation they're for. Why is that a good thing to do? It seems
like such Gathers are necessarily being made with incomplete information,
and we'd be better off to fix things so that none are made till later.
regards, tom lane
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
@ 2018-12-29 01:05 ` Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2018-12-29 01:05 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
>
> Kohei KaiGai <[email protected]> writes:
> > I've investigated a crash report of PG-Strom for a few days, then I doubt
> > add_partial_path() can unexpectedly release dominated old partial path
> > but still referenced by other Gather node, and it leads unexpected system
> > crash.
>
> Hm. This seems comparable to the special case in plain add_path, where it
> doesn't attempt to free IndexPaths because of the risk that they're still
> referenced. So maybe we should just drop the pfree here.
>
> However, first I'd like to know why this situation is arising in the first
> place. To have the situation you're describing, we'd have to have
> attempted to make some Gather paths before we have all the partial paths
> for the relation they're for. Why is that a good thing to do? It seems
> like such Gathers are necessarily being made with incomplete information,
> and we'd be better off to fix things so that none are made till later.
>
Because of the hook location, Gather-node shall be constructed with built-in
and foreign partial scan node first, then extension gets a chance to add its
custom paths (partial and full).
At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
generate_gather_paths(). Even if extension adds some partial paths later,
the first generate_gather_paths() has to generate Gather node based on
incomplete information.
If we could ensure Gather node shall be made after all the partial nodes
are added, it may be a solution for the problem.
Of course, relocation of the hook may have a side-effect. Anyone may
expect the pathlist contains all the path-node including Gather node, for
editorialization of the pathlist.
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2018-12-29 19:12 ` Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Tom Lane @ 2018-12-29 19:12 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: pgsql-hackers
Kohei KaiGai <[email protected]> writes:
> 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
>> However, first I'd like to know why this situation is arising in the first
>> place. To have the situation you're describing, we'd have to have
>> attempted to make some Gather paths before we have all the partial paths
>> for the relation they're for. Why is that a good thing to do? It seems
>> like such Gathers are necessarily being made with incomplete information,
>> and we'd be better off to fix things so that none are made till later.
> Because of the hook location, Gather-node shall be constructed with built-in
> and foreign partial scan node first, then extension gets a chance to add its
> custom paths (partial and full).
> At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> generate_gather_paths().
Hmm. I'm inclined to think that we should have a separate hook
in which extensions are allowed to add partial paths, and that
set_rel_pathlist_hook should only be allowed to add regular paths.
regards, tom lane
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
@ 2018-12-30 03:31 ` Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Kohei KaiGai @ 2018-12-30 03:31 UTC (permalink / raw)
To: Tom Lane <[email protected]>; [email protected]; +Cc: pgsql-hackers
2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
>
> Kohei KaiGai <[email protected]> writes:
> > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> >> However, first I'd like to know why this situation is arising in the first
> >> place. To have the situation you're describing, we'd have to have
> >> attempted to make some Gather paths before we have all the partial paths
> >> for the relation they're for. Why is that a good thing to do? It seems
> >> like such Gathers are necessarily being made with incomplete information,
> >> and we'd be better off to fix things so that none are made till later.
>
> > Because of the hook location, Gather-node shall be constructed with built-in
> > and foreign partial scan node first, then extension gets a chance to add its
> > custom paths (partial and full).
> > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > generate_gather_paths().
>
> Hmm. I'm inclined to think that we should have a separate hook
> in which extensions are allowed to add partial paths, and that
> set_rel_pathlist_hook should only be allowed to add regular paths.
>
I have almost same opinion, but the first hook does not need to be
dedicated for partial paths. As like set_foreign_pathlist() doing, we can
add both of partial and regular paths here, then generate_gather_paths()
may generate a Gather-path on top of the best partial-path.
On the other hands, the later hook must be dedicated to add regular paths,
and also provides a chance for extensions to manipulate pre-built path-list
including Gather-path.
As long as I know, pg_hint_plan uses the set_rel_pathlist_hook to enforce
a particular path-node, including Gather-node, by manipulation of the cost
value. Horiguchi-san, is it right?
Likely, this kind of extension needs to use the later hook.
I expect these hooks are located as follows:
set_rel_pathlist(...)
{
:
<snip>
:
/* for partial / regular paths */
if (set_rel_pathlist_hook)
(*set_rel_pathlist_hook) (root, rel, rti, rte);
/* generate Gather-node */
if (rel->reloptkind == RELOPT_BASEREL)
generate_gather_paths(root, rel);
/* for regular paths and manipulation */
if (post_rel_pathlist_hook)
(*post_rel_pathlist_hook) (root, rel, rti, rte);
set_cheapest();
}
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2018-12-31 04:09 ` Amit Kapila <[email protected]>
2018-12-31 12:17 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Amit Kapila @ 2018-12-31 04:09 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Tom Lane <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai <[email protected]> wrote:
> 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> >
> > Kohei KaiGai <[email protected]> writes:
> > > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> > >> However, first I'd like to know why this situation is arising in the first
> > >> place. To have the situation you're describing, we'd have to have
> > >> attempted to make some Gather paths before we have all the partial paths
> > >> for the relation they're for. Why is that a good thing to do? It seems
> > >> like such Gathers are necessarily being made with incomplete information,
> > >> and we'd be better off to fix things so that none are made till later.
> >
> > > Because of the hook location, Gather-node shall be constructed with built-in
> > > and foreign partial scan node first, then extension gets a chance to add its
> > > custom paths (partial and full).
> > > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > > generate_gather_paths().
> >
> > Hmm. I'm inclined to think that we should have a separate hook
> > in which extensions are allowed to add partial paths, and that
> > set_rel_pathlist_hook should only be allowed to add regular paths.
+1. This idea sounds sensible to me.
> >
> I have almost same opinion, but the first hook does not need to be
> dedicated for partial paths. As like set_foreign_pathlist() doing, we can
> add both of partial and regular paths here, then generate_gather_paths()
> may generate a Gather-path on top of the best partial-path.
>
Won't it be confusing for users if we allow both partial and full
paths in first hook and only full paths in the second hook?
Basically, in many cases, the second hook won't be of much use. What
advantage you are seeing in allowing both partial and full paths in
the first hook?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
@ 2018-12-31 12:17 ` Kohei KaiGai <[email protected]>
2018-12-31 13:25 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2018-12-31 12:17 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tom Lane <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
2018年12月31日(月) 13:10 Amit Kapila <[email protected]>:
>
> On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai <[email protected]> wrote:
> > 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> > >
> > > Kohei KaiGai <[email protected]> writes:
> > > > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> > > >> However, first I'd like to know why this situation is arising in the first
> > > >> place. To have the situation you're describing, we'd have to have
> > > >> attempted to make some Gather paths before we have all the partial paths
> > > >> for the relation they're for. Why is that a good thing to do? It seems
> > > >> like such Gathers are necessarily being made with incomplete information,
> > > >> and we'd be better off to fix things so that none are made till later.
> > >
> > > > Because of the hook location, Gather-node shall be constructed with built-in
> > > > and foreign partial scan node first, then extension gets a chance to add its
> > > > custom paths (partial and full).
> > > > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > > > generate_gather_paths().
> > >
> > > Hmm. I'm inclined to think that we should have a separate hook
> > > in which extensions are allowed to add partial paths, and that
> > > set_rel_pathlist_hook should only be allowed to add regular paths.
>
> +1. This idea sounds sensible to me.
>
> > >
> > I have almost same opinion, but the first hook does not need to be
> > dedicated for partial paths. As like set_foreign_pathlist() doing, we can
> > add both of partial and regular paths here, then generate_gather_paths()
> > may generate a Gather-path on top of the best partial-path.
> >
>
> Won't it be confusing for users if we allow both partial and full
> paths in first hook and only full paths in the second hook?
> Basically, in many cases, the second hook won't be of much use. What
> advantage you are seeing in allowing both partial and full paths in
> the first hook?
>
Two advantages. The first one is, it follows same manner of
set_foreign_pathlist()
which allows to add both of full and partial path if FDW supports parallel-scan.
The second one is practical. During the path construction, extension needs to
check availability to run (e.g, whether operators in WHERE-clause is supported
on GPU device...), calculate its estimated cost and so on. Not a small
portion of
them are common for both of full and partial path. So, if the first
hook accepts to
add both kind of paths at once, extension can share the common properties.
Probably, the second hook is only used for a few corner case where an extension
wants to manipulate path-list already built, like pg_hint_plan.
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
2018-12-31 12:17 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2018-12-31 13:25 ` Amit Kapila <[email protected]>
2019-01-02 13:34 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Amit Kapila @ 2018-12-31 13:25 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Tom Lane <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Mon, Dec 31, 2018 at 5:48 PM Kohei KaiGai <[email protected]> wrote:
>
> 2018年12月31日(月) 13:10 Amit Kapila <[email protected]>:
> >
> > On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai <[email protected]> wrote:
> > > 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> > > >
> > > > Kohei KaiGai <[email protected]> writes:
> > > > > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> > > > >> However, first I'd like to know why this situation is arising in the first
> > > > >> place. To have the situation you're describing, we'd have to have
> > > > >> attempted to make some Gather paths before we have all the partial paths
> > > > >> for the relation they're for. Why is that a good thing to do? It seems
> > > > >> like such Gathers are necessarily being made with incomplete information,
> > > > >> and we'd be better off to fix things so that none are made till later.
> > > >
> > > > > Because of the hook location, Gather-node shall be constructed with built-in
> > > > > and foreign partial scan node first, then extension gets a chance to add its
> > > > > custom paths (partial and full).
> > > > > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > > > > generate_gather_paths().
> > > >
> > > > Hmm. I'm inclined to think that we should have a separate hook
> > > > in which extensions are allowed to add partial paths, and that
> > > > set_rel_pathlist_hook should only be allowed to add regular paths.
> >
> > +1. This idea sounds sensible to me.
> >
> > > >
> > > I have almost same opinion, but the first hook does not need to be
> > > dedicated for partial paths. As like set_foreign_pathlist() doing, we can
> > > add both of partial and regular paths here, then generate_gather_paths()
> > > may generate a Gather-path on top of the best partial-path.
> > >
> >
> > Won't it be confusing for users if we allow both partial and full
> > paths in first hook and only full paths in the second hook?
> > Basically, in many cases, the second hook won't be of much use. What
> > advantage you are seeing in allowing both partial and full paths in
> > the first hook?
> >
> Two advantages. The first one is, it follows same manner of
> set_foreign_pathlist()
> which allows to add both of full and partial path if FDW supports parallel-scan.
> The second one is practical. During the path construction, extension needs to
> check availability to run (e.g, whether operators in WHERE-clause is supported
> on GPU device...), calculate its estimated cost and so on. Not a small
> portion of
> them are common for both of full and partial path. So, if the first
> hook accepts to
> add both kind of paths at once, extension can share the common properties.
>
You have a point, though I am not sure how much difference it can
create for cost computation as ideally, both will have different
costing model. I understand there are some savings by avoiding some
common work, is there any way to cache the required information?
> Probably, the second hook is only used for a few corner case where an extension
> wants to manipulate path-list already built, like pg_hint_plan.
>
Okay, but it could be some work for extension authors who are using
the current hook, not sure they would like to divide the work between
first and second hook.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
2018-12-31 12:17 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 13:25 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
@ 2019-01-02 13:34 ` Kohei KaiGai <[email protected]>
2019-01-04 04:46 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2019-01-02 13:34 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tom Lane <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
2018年12月31日(月) 22:25 Amit Kapila <[email protected]>:
>
> On Mon, Dec 31, 2018 at 5:48 PM Kohei KaiGai <[email protected]> wrote:
> >
> > 2018年12月31日(月) 13:10 Amit Kapila <[email protected]>:
> > >
> > > On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai <[email protected]> wrote:
> > > > 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> > > > >
> > > > > Kohei KaiGai <[email protected]> writes:
> > > > > > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> > > > > >> However, first I'd like to know why this situation is arising in the first
> > > > > >> place. To have the situation you're describing, we'd have to have
> > > > > >> attempted to make some Gather paths before we have all the partial paths
> > > > > >> for the relation they're for. Why is that a good thing to do? It seems
> > > > > >> like such Gathers are necessarily being made with incomplete information,
> > > > > >> and we'd be better off to fix things so that none are made till later.
> > > > >
> > > > > > Because of the hook location, Gather-node shall be constructed with built-in
> > > > > > and foreign partial scan node first, then extension gets a chance to add its
> > > > > > custom paths (partial and full).
> > > > > > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > > > > > generate_gather_paths().
> > > > >
> > > > > Hmm. I'm inclined to think that we should have a separate hook
> > > > > in which extensions are allowed to add partial paths, and that
> > > > > set_rel_pathlist_hook should only be allowed to add regular paths.
> > >
> > > +1. This idea sounds sensible to me.
> > >
> > > > >
> > > > I have almost same opinion, but the first hook does not need to be
> > > > dedicated for partial paths. As like set_foreign_pathlist() doing, we can
> > > > add both of partial and regular paths here, then generate_gather_paths()
> > > > may generate a Gather-path on top of the best partial-path.
> > > >
> > >
> > > Won't it be confusing for users if we allow both partial and full
> > > paths in first hook and only full paths in the second hook?
> > > Basically, in many cases, the second hook won't be of much use. What
> > > advantage you are seeing in allowing both partial and full paths in
> > > the first hook?
> > >
> > Two advantages. The first one is, it follows same manner of
> > set_foreign_pathlist()
> > which allows to add both of full and partial path if FDW supports parallel-scan.
> > The second one is practical. During the path construction, extension needs to
> > check availability to run (e.g, whether operators in WHERE-clause is supported
> > on GPU device...), calculate its estimated cost and so on. Not a small
> > portion of
> > them are common for both of full and partial path. So, if the first
> > hook accepts to
> > add both kind of paths at once, extension can share the common properties.
> >
>
> You have a point, though I am not sure how much difference it can
> create for cost computation as ideally, both will have different
> costing model. I understand there are some savings by avoiding some
> common work, is there any way to cache the required information?
>
I have no idea for the clean way.
We may be able to have an opaque pointer for extension usage, however,
it may be problematic if multiple extension uses the hook.
> > Probably, the second hook is only used for a few corner case where an extension
> > wants to manipulate path-list already built, like pg_hint_plan.
> >
>
> Okay, but it could be some work for extension authors who are using
> the current hook, not sure they would like to divide the work between
> first and second hook.
>
I guess they don't divide their code, but choose either of them.
In case of PG-Strom, even if there are two hooks around the point, it will use
the first hook only, unless it does not prohibit to call add_path() here.
However, some adjustments are required. Its current implementation makes
GatherPath node with partial CustomScanPath because set_rel_pathlist_hook()
is called after the generate_gather_paths().
Once we could choose the first hook, no need to make a GatherPath by itself,
because PostgreSQL-core will make the path if partial custom-path is enough
reasonable cost. Likely, this adjustment is more preferable one.
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
2018-12-31 12:17 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-31 13:25 ` Re: add_partial_path() may remove dominated path but still in use Amit Kapila <[email protected]>
2019-01-02 13:34 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2019-01-04 04:46 ` Kohei KaiGai <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Kohei KaiGai @ 2019-01-04 04:46 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tom Lane <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
I tried to make a patch to have dual hooks at set_rel_pathlist(), and
adjusted PG-Strom for the new design. It stopped to create GatherPath
by itself, just added a partial path for the base relation.
It successfully made a plan using parallel custom-scan node, without
system crash.
As I mentioned above, it does not use the new "post_rel_pathlist_hook"
because we can add both of partial/regular path-node at the first hook
with no particular problems.
Thanks,
dbt3=# explain select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < cast(date '1994-01-01' + interval '1 year' as date)
and l_discount between 0.08 - 0.01 and 0.08 + 0.01
and l_quantity < 24 limit 1;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Limit (cost=144332.62..144332.63 rows=1 width=4)
-> Aggregate (cost=144332.62..144332.63 rows=1 width=4)
-> Gather (cost=144285.70..144329.56 rows=408 width=4)
Workers Planned: 2
-> Parallel Custom Scan (GpuPreAgg) on lineitem
(cost=143285.70..143288.76 rows=204 width=4)
Reduction: NoGroup
Outer Scan: lineitem (cost=1666.67..143246.16
rows=63254 width=8)
Outer Scan Filter: ((l_discount >= '0.07'::double
precision) AND
(l_discount <=
'0.09'::double precision) AND
(l_quantity <
'24'::double precision) AND
(l_shipdate <
'1995-01-01'::date) AND
(l_shipdate >=
'1994-01-01'::date))
(8 rows)
Thanks,
2019年1月2日(水) 22:34 Kohei KaiGai <[email protected]>:
>
> 2018年12月31日(月) 22:25 Amit Kapila <[email protected]>:
> >
> > On Mon, Dec 31, 2018 at 5:48 PM Kohei KaiGai <[email protected]> wrote:
> > >
> > > 2018年12月31日(月) 13:10 Amit Kapila <[email protected]>:
> > > >
> > > > On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai <[email protected]> wrote:
> > > > > 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> > > > > >
> > > > > > Kohei KaiGai <[email protected]> writes:
> > > > > > > 2018年12月29日(土) 1:44 Tom Lane <[email protected]>:
> > > > > > >> However, first I'd like to know why this situation is arising in the first
> > > > > > >> place. To have the situation you're describing, we'd have to have
> > > > > > >> attempted to make some Gather paths before we have all the partial paths
> > > > > > >> for the relation they're for. Why is that a good thing to do? It seems
> > > > > > >> like such Gathers are necessarily being made with incomplete information,
> > > > > > >> and we'd be better off to fix things so that none are made till later.
> > > > > >
> > > > > > > Because of the hook location, Gather-node shall be constructed with built-in
> > > > > > > and foreign partial scan node first, then extension gets a chance to add its
> > > > > > > custom paths (partial and full).
> > > > > > > At the set_rel_pathlist(), set_rel_pathlist_hook() is invoked next to the
> > > > > > > generate_gather_paths().
> > > > > >
> > > > > > Hmm. I'm inclined to think that we should have a separate hook
> > > > > > in which extensions are allowed to add partial paths, and that
> > > > > > set_rel_pathlist_hook should only be allowed to add regular paths.
> > > >
> > > > +1. This idea sounds sensible to me.
> > > >
> > > > > >
> > > > > I have almost same opinion, but the first hook does not need to be
> > > > > dedicated for partial paths. As like set_foreign_pathlist() doing, we can
> > > > > add both of partial and regular paths here, then generate_gather_paths()
> > > > > may generate a Gather-path on top of the best partial-path.
> > > > >
> > > >
> > > > Won't it be confusing for users if we allow both partial and full
> > > > paths in first hook and only full paths in the second hook?
> > > > Basically, in many cases, the second hook won't be of much use. What
> > > > advantage you are seeing in allowing both partial and full paths in
> > > > the first hook?
> > > >
> > > Two advantages. The first one is, it follows same manner of
> > > set_foreign_pathlist()
> > > which allows to add both of full and partial path if FDW supports parallel-scan.
> > > The second one is practical. During the path construction, extension needs to
> > > check availability to run (e.g, whether operators in WHERE-clause is supported
> > > on GPU device...), calculate its estimated cost and so on. Not a small
> > > portion of
> > > them are common for both of full and partial path. So, if the first
> > > hook accepts to
> > > add both kind of paths at once, extension can share the common properties.
> > >
> >
> > You have a point, though I am not sure how much difference it can
> > create for cost computation as ideally, both will have different
> > costing model. I understand there are some savings by avoiding some
> > common work, is there any way to cache the required information?
> >
> I have no idea for the clean way.
> We may be able to have an opaque pointer for extension usage, however,
> it may be problematic if multiple extension uses the hook.
>
> > > Probably, the second hook is only used for a few corner case where an extension
> > > wants to manipulate path-list already built, like pg_hint_plan.
> > >
> >
> > Okay, but it could be some work for extension authors who are using
> > the current hook, not sure they would like to divide the work between
> > first and second hook.
> >
> I guess they don't divide their code, but choose either of them.
> In case of PG-Strom, even if there are two hooks around the point, it will use
> the first hook only, unless it does not prohibit to call add_path() here.
> However, some adjustments are required. Its current implementation makes
> GatherPath node with partial CustomScanPath because set_rel_pathlist_hook()
> is called after the generate_gather_paths().
> Once we could choose the first hook, no need to make a GatherPath by itself,
> because PostgreSQL-core will make the path if partial custom-path is enough
> reasonable cost. Likely, this adjustment is more preferable one.
>
> Thanks,
> --
> HeteroDB, Inc / The PG-Strom Project
> KaiGai Kohei <[email protected]>
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
Attachments:
[application/octet-stream] pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v10.patch (2.8K, ../../CAOP8fzZYvQWpANOzF4acMgOee7T6p9m12a2ZKfhPv5nCXtFLZw@mail.gmail.com/2-pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v10.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 22 +++++++++++++++++-----
src/include/optimizer/paths.h | 1 +
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index d9a2f9b..100274e 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -62,6 +62,7 @@ int min_parallel_index_scan_size;
/* Hook for plugins to get control in set_rel_pathlist() */
set_rel_pathlist_hook_type set_rel_pathlist_hook = NULL;
+set_rel_pathlist_hook_type post_rel_pathlist_hook = NULL;
/* Hook for plugins to replace standard_join_search() */
join_search_hook_type join_search_hook = NULL;
@@ -479,6 +480,14 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * Allow a plugin to editorialize on the set of Paths for this base
+ * relation. It could add new regular paths (such as CustomPaths) by
+ * calling add_path(), partial paths by add_partial_path().
+ */
+ if (set_rel_pathlist_hook)
+ (*set_rel_pathlist_hook) (root, rel, rti, rte);
+
+ /*
* If this is a baserel, consider gathering any partial paths we may have
* created for it. (If we tried to gather inheritance children, we could
* end up with a very large number of gather nodes, each trying to grab
@@ -489,12 +498,15 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
generate_gather_paths(root, rel);
/*
- * Allow a plugin to editorialize on the set of Paths for this base
- * relation. It could add new paths (such as CustomPaths) by calling
- * add_path(), or delete or modify paths added by the core code.
+ * Allow a plugin to manipulate all the set of Paths for this base
+ * relation. It could delete or modify paths added by the core or
+ * extensions at the prior hook.
+ * Note that we don't recommend to add partial paths here, because
+ * GatherPath is already built above, and alternative cheaper partial
+ * path may release path-node in use.
*/
- if (set_rel_pathlist_hook)
- (*set_rel_pathlist_hook) (root, rel, rti, rte);
+ if (post_rel_pathlist_hook)
+ (*post_rel_pathlist_hook) (root, rel, rti, rte);
/* Now find the cheapest of the paths for this rel */
set_cheapest(rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 1d3fb5c..1950775 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -31,6 +31,7 @@ typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
Index rti,
RangeTblEntry *rte);
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;
+extern PGDLLIMPORT set_rel_pathlist_hook_type post_rel_pathlist_hook;
/* Hook for plugins to get control in add_paths_to_joinrel() */
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
[application/octet-stream] pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v11.patch (2.7K, ../../CAOP8fzZYvQWpANOzF4acMgOee7T6p9m12a2ZKfhPv5nCXtFLZw@mail.gmail.com/3-pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v11.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 22 +++++++++++++++++-----
src/include/optimizer/paths.h | 1 +
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 5f74d3b..757249b 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -64,6 +64,7 @@ int min_parallel_index_scan_size;
/* Hook for plugins to get control in set_rel_pathlist() */
set_rel_pathlist_hook_type set_rel_pathlist_hook = NULL;
+set_rel_pathlist_hook_type post_rel_pathlist_hook = NULL;
/* Hook for plugins to replace standard_join_search() */
join_search_hook_type join_search_hook = NULL;
@@ -480,6 +481,14 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * Allow a plugin to editorialize on the set of Paths for this base
+ * relation. It could add new regular paths (such as CustomPaths) by
+ * calling add_path(), partial paths by add_partial_path().
+ */
+ if (set_rel_pathlist_hook)
+ (*set_rel_pathlist_hook) (root, rel, rti, rte);
+
+ /*
* If this is a baserel, we should normally consider gathering any partial
* paths we may have created for it.
*
@@ -497,12 +506,15 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
generate_gather_paths(root, rel, false);
/*
- * Allow a plugin to editorialize on the set of Paths for this base
- * relation. It could add new paths (such as CustomPaths) by calling
- * add_path(), or delete or modify paths added by the core code.
+ * Allow a plugin to manipulate all the set of Paths for this base
+ * relation. It could delete or modify paths added by the core or
+ * extensions at the prior hook.
+ * Note that we don't recommend to add partial paths here, because
+ * GatherPath is already built above, and alternative cheaper partial
+ * path may release path-node in use.
*/
- if (set_rel_pathlist_hook)
- (*set_rel_pathlist_hook) (root, rel, rti, rte);
+ if (post_rel_pathlist_hook)
+ (*post_rel_pathlist_hook) (root, rel, rti, rte);
/* Now find the cheapest of the paths for this rel */
set_cheapest(rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index cafde30..74bd9c3 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -31,6 +31,7 @@ typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
Index rti,
RangeTblEntry *rte);
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;
+extern PGDLLIMPORT set_rel_pathlist_hook_type post_rel_pathlist_hook;
/* Hook for plugins to get control in add_paths_to_joinrel() */
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2019-01-09 04:18 ` Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-01-09 04:18 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
At Sun, 30 Dec 2018 12:31:22 +0900, Kohei KaiGai <[email protected]> wrote in <CAOP8fzY1Oqf-LGdrZT+TAu+JajwPGn1uYnpWWUPL=2LiattjYA@mail.gmail.com>
> 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> On the other hands, the later hook must be dedicated to add regular paths,
> and also provides a chance for extensions to manipulate pre-built path-list
> including Gather-path.
> As long as I know, pg_hint_plan uses the set_rel_pathlist_hook to enforce
> a particular path-node, including Gather-node, by manipulation of the cost
> value. Horiguchi-san, is it right?
Mmm. I haven't expected that it is mentioned here.
Actually in the hook, it changes enable_* planner variables, or
directory manipuraltes path costs or even can clear and
regenerate the path list and gather paths for the parallel
case. It will be happy if we had a chance to manpurate partial
paths before genrating gahter paths.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
@ 2019-01-09 05:44 ` Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2019-01-09 05:44 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
2019年1月9日(水) 13:18 Kyotaro HORIGUCHI <[email protected]>:
>
> At Sun, 30 Dec 2018 12:31:22 +0900, Kohei KaiGai <[email protected]> wrote in <CAOP8fzY1Oqf-LGdrZT+TAu+JajwPGn1uYnpWWUPL=2LiattjYA@mail.gmail.com>
> > 2018年12月30日(日) 4:12 Tom Lane <[email protected]>:
> > On the other hands, the later hook must be dedicated to add regular paths,
> > and also provides a chance for extensions to manipulate pre-built path-list
> > including Gather-path.
> > As long as I know, pg_hint_plan uses the set_rel_pathlist_hook to enforce
> > a particular path-node, including Gather-node, by manipulation of the cost
> > value. Horiguchi-san, is it right?
>
> Mmm. I haven't expected that it is mentioned here.
>
> Actually in the hook, it changes enable_* planner variables, or
> directory manipuraltes path costs or even can clear and
> regenerate the path list and gather paths for the parallel
> case. It will be happy if we had a chance to manpurate partial
> paths before genrating gahter paths.
>
So, is it sufficient if set_rel_pathlist_hook is just relocated in
front of the generate_gather_paths?
If we have no use case for the second hook, here is little necessity
to have the post_rel_pathlist_hook() here.
(At least, PG-Strom will use the first hook only.)
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2019-01-10 20:52 ` Robert Haas <[email protected]>
2019-01-11 02:09 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Robert Haas @ 2019-01-10 20:52 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai <[email protected]> wrote:
> So, is it sufficient if set_rel_pathlist_hook is just relocated in
> front of the generate_gather_paths?
> If we have no use case for the second hook, here is little necessity
> to have the post_rel_pathlist_hook() here.
> (At least, PG-Strom will use the first hook only.)
+1. That seems like the best way to be consistent with the principle
that we need to have all the partial paths before generating any
Gather paths.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
@ 2019-01-11 02:09 ` Kohei KaiGai <[email protected]>
2019-01-11 16:36 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kohei KaiGai @ 2019-01-11 02:09 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
2019年1月11日(金) 5:52 Robert Haas <[email protected]>:
>
> On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai <[email protected]> wrote:
> > So, is it sufficient if set_rel_pathlist_hook is just relocated in
> > front of the generate_gather_paths?
> > If we have no use case for the second hook, here is little necessity
> > to have the post_rel_pathlist_hook() here.
> > (At least, PG-Strom will use the first hook only.)
>
> +1. That seems like the best way to be consistent with the principle
> that we need to have all the partial paths before generating any
> Gather paths.
>
Patch was updated, just for relocation of the set_rel_pathlist_hook.
Please check it.
Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
Attachments:
[application/octet-stream] pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v10.patch (1.5K, ../../CAOP8fzZN9JwT466Lp-WYv9wGHZoPfG_qS2_fJ0wxiVKxitZwkw@mail.gmail.com/2-pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v10.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index d9a2f9b..9a44c3b 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -479,6 +479,15 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * Allow a plugin to editorialize on the set of Paths for this base
+ * relation. It could add new paths (such as CustomPaths) by calling
+ * add_path(), or add_partial_path() if parallel aware, or delete or
+ * modify paths added by the core code.
+ */
+ if (set_rel_pathlist_hook)
+ (*set_rel_pathlist_hook) (root, rel, rti, rte);
+
+ /*
* If this is a baserel, consider gathering any partial paths we may have
* created for it. (If we tried to gather inheritance children, we could
* end up with a very large number of gather nodes, each trying to grab
@@ -488,14 +497,6 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
if (rel->reloptkind == RELOPT_BASEREL)
generate_gather_paths(root, rel);
- /*
- * Allow a plugin to editorialize on the set of Paths for this base
- * relation. It could add new paths (such as CustomPaths) by calling
- * add_path(), or delete or modify paths added by the core code.
- */
- if (set_rel_pathlist_hook)
- (*set_rel_pathlist_hook) (root, rel, rti, rte);
-
/* Now find the cheapest of the paths for this rel */
set_cheapest(rel);
[application/octet-stream] pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v11.patch (1.4K, ../../CAOP8fzZN9JwT466Lp-WYv9wGHZoPfG_qS2_fJ0wxiVKxitZwkw@mail.gmail.com/3-pgsql-bugfix-relocation-path-set_rel_pathlist_hook.v11.patch)
download | inline diff:
src/backend/optimizer/path/allpaths.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 5f74d3b..6f73dcd 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -480,6 +480,15 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * Allow a plugin to editorialize on the set of Paths for this base
+ * relation. It could add new paths (such as CustomPaths) by calling
+ * add_path(), or add_partial_path() if parallel aware, or delete or
+ * modify paths added by the core code.
+ */
+ if (set_rel_pathlist_hook)
+ (*set_rel_pathlist_hook) (root, rel, rti, rte);
+
+ /*
* If this is a baserel, we should normally consider gathering any partial
* paths we may have created for it.
*
@@ -496,14 +505,6 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
bms_membership(root->all_baserels) != BMS_SINGLETON)
generate_gather_paths(root, rel, false);
- /*
- * Allow a plugin to editorialize on the set of Paths for this base
- * relation. It could add new paths (such as CustomPaths) by calling
- * add_path(), or delete or modify paths added by the core code.
- */
- if (set_rel_pathlist_hook)
- (*set_rel_pathlist_hook) (root, rel, rti, rte);
-
/* Now find the cheapest of the paths for this rel */
set_cheapest(rel);
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
2019-01-11 02:09 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
@ 2019-01-11 16:36 ` Robert Haas <[email protected]>
2019-01-17 09:28 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Robert Haas @ 2019-01-11 16:36 UTC (permalink / raw)
To: Kohei KaiGai <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
On Thu, Jan 10, 2019 at 9:10 PM Kohei KaiGai <[email protected]> wrote:
> 2019年1月11日(金) 5:52 Robert Haas <[email protected]>:
> > On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai <[email protected]> wrote:
> > > So, is it sufficient if set_rel_pathlist_hook is just relocated in
> > > front of the generate_gather_paths?
> > > If we have no use case for the second hook, here is little necessity
> > > to have the post_rel_pathlist_hook() here.
> > > (At least, PG-Strom will use the first hook only.)
> >
> > +1. That seems like the best way to be consistent with the principle
> > that we need to have all the partial paths before generating any
> > Gather paths.
> >
> Patch was updated, just for relocation of the set_rel_pathlist_hook.
> Please check it.
Seems reasonable to me.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
2019-01-11 02:09 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-11 16:36 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
@ 2019-01-17 09:28 ` Kyotaro HORIGUCHI <[email protected]>
2019-01-22 11:50 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-01-17 09:28 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; pgsql-hackers
Hello, sorry for the absence.
At Fri, 11 Jan 2019 11:36:43 -0500, Robert Haas <[email protected]> wrote in <CA+TgmoYyxBgkfN_APBdxdutFMukb=P-EgGNY-NbauRcL7mGnmA@mail.gmail.com>
> On Thu, Jan 10, 2019 at 9:10 PM Kohei KaiGai <[email protected]> wrote:
> > 2019年1月11日(金) 5:52 Robert Haas <[email protected]>:
> > > On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai <[email protected]> wrote:
> > > > So, is it sufficient if set_rel_pathlist_hook is just relocated in
> > > > front of the generate_gather_paths?
> > > > If we have no use case for the second hook, here is little necessity
> > > > to have the post_rel_pathlist_hook() here.
> > > > (At least, PG-Strom will use the first hook only.)
> > >
> > > +1. That seems like the best way to be consistent with the principle
> > > that we need to have all the partial paths before generating any
> > > Gather paths.
> > >
> > Patch was updated, just for relocation of the set_rel_pathlist_hook.
> > Please check it.
>
> Seems reasonable to me.
Also seems reasonable to me. The extension can call
generate_gather_paths redundantly as is but it almost doesn't
harm, so it is acceptable even in a minor release.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: add_partial_path() may remove dominated path but still in use
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-29 01:05 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Re: add_partial_path() may remove dominated path but still in use Tom Lane <[email protected]>
2018-12-30 03:31 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
2019-01-11 02:09 ` Re: add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2019-01-11 16:36 ` Re: add_partial_path() may remove dominated path but still in use Robert Haas <[email protected]>
2019-01-17 09:28 ` Re: add_partial_path() may remove dominated path but still in use Kyotaro HORIGUCHI <[email protected]>
@ 2019-01-22 11:50 ` Kohei KaiGai <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Kohei KaiGai @ 2019-01-22 11:50 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers
Let me remind the thread.
If no more comments, objections, or better ideas, please commit this fix.
Thanks,
2019年1月17日(木) 18:29 Kyotaro HORIGUCHI <[email protected]>:
>
> Hello, sorry for the absence.
>
> At Fri, 11 Jan 2019 11:36:43 -0500, Robert Haas <[email protected]> wrote in <CA+TgmoYyxBgkfN_APBdxdutFMukb=P-EgGNY-NbauRcL7mGnmA@mail.gmail.com>
> > On Thu, Jan 10, 2019 at 9:10 PM Kohei KaiGai <[email protected]> wrote:
> > > 2019年1月11日(金) 5:52 Robert Haas <[email protected]>:
> > > > On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai <[email protected]> wrote:
> > > > > So, is it sufficient if set_rel_pathlist_hook is just relocated in
> > > > > front of the generate_gather_paths?
> > > > > If we have no use case for the second hook, here is little necessity
> > > > > to have the post_rel_pathlist_hook() here.
> > > > > (At least, PG-Strom will use the first hook only.)
> > > >
> > > > +1. That seems like the best way to be consistent with the principle
> > > > that we need to have all the partial paths before generating any
> > > > Gather paths.
> > > >
> > > Patch was updated, just for relocation of the set_rel_pathlist_hook.
> > > Please check it.
> >
> > Seems reasonable to me.
>
> Also seems reasonable to me. The extension can call
> generate_gather_paths redundantly as is but it almost doesn't
> harm, so it is acceptable even in a minor release.
>
> regards.
>
> --
> Kyotaro Horiguchi
> NTT Open Source Software Center
>
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
@ 2023-05-03 21:59 Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 21:57 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-03 21:59 UTC (permalink / raw)
To: samay sharma <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi Samay,
On Tue, May 2, 2023 at 11:40 PM samay sharma <[email protected]> wrote:
> Thanks for taking the time to do this. It is indeed difficult work.
Thanks for the review! I think that this is something that would
definitely benefit from a perspective such as yours.
> There are things I like about the changes you've proposed and some where I feel that the previous section was easier to understand.
That makes sense, and I think that I agree with every point you've
raised, bar none. I'm pleased to see that you basically agree with the
high level direction.
I would estimate that the version you looked at (v2) is perhaps 35%
complete. So some of the individual problems you noticed were a direct
consequence of the work just not being anywhere near complete. I'll
try to do a better job of tracking the relative maturity of each
commit/patch in each commit message, going forward.
Anything that falls under "25.2.1. Recovering Disk Space" is
particularly undeveloped in v2. The way that I broke that up into a
bunch of WARNINGs/NOTEs/TIPs was just a short term way of breaking it
up into pieces, so that the structure was very approximately what I
wanted. I actually think that the stuff about CLUSTER and VACUUM FULL
belongs in a completely different chapter. Since it is not "Routine
Vacuuming" at all.
>> 2. Renamed "Preventing Transaction ID Wraparound Failures" to
>> "Freezing to manage the transaction ID space". Now we talk about
>> wraparound as a subtopic of freezing, not vice-versa. (This is a
>> complete rewrite, as described by later items in this list).
>
> +1 on this too. Freezing is a normal part of vacuuming and while the aggressive vacuums are different, I think just talking about the worst case scenario while referring to it is alarmist.
Strangely enough, Postgres 16 is the first version that instruments
freezing in its autovacuum log reports. I suspect that some long term
users will find it quite surprising to see how much (or how little)
freezing takes place in non-aggressive VACUUMs.
The introduction of page-level freezing will make it easier and more
natural to tune settings like vacuum_freeze_min_age, with the aim of
smoothing out the burden of freezing over time (particularly by making
non-aggressive VACUUMs freeze more). Page-level freezing removes any
question of not freezing every tuple on a page (barring cases where
"removable cutoff" is noticeably held back by an old MVCC snapshot).
This makes it more natural to think of freezing as a process that
makes it okay to store data in individual physical heap pages, long
term.
> 1) While I agree that bundling VACUUM and VACUUM FULL is not the right way, moving all VACUUM FULL references into tips and warnings also seems excessive. I think it's probably best to just have a single paragraph which talks about VACUUM FULL as I do think it should be mentioned in the reclaiming disk space section.
As I mentioned briefly already, my intention is to move it to another
chapter entirely. I was thinking of "Chapter 29. Monitoring Disk
Usage". The "Routine Vacuuming" docs would then link to this sect1 --
something along the lines of "non-routine commands to reclaim a lot of
disk space in the event of extreme bloat".
> 2) I felt that the new section, "Freezing to manage the transaction ID space" could be made simpler to understand. As an example, I understood what the parameters (autovacuum_freeze_max_age, vacuum_freeze_table_age) do and how they interact better in the previous version of the docs.
Agreed. I'm going to split it up some more. I think that the current
"25.2.2.1. VACUUM's Aggressive Strategy" should be split in two, so we
go from talking about aggressive VACUUMs to Antiwraparound
autovacuums. Finding the least confusing way of explaining it has been
a focus of mine in the last few days.
> 4) I think we should explicitly call out that seeing an anti-wraparound VACUUM or "VACUUM table (to prevent wraparound)" is normal and that it's just a VACUUM triggered due to the table having unfrozen rows with an XID older than autovacuum_freeze_max_age. I've seen many users panicking on seeing this and feeling that they are close to a wraparound.
That has also been my exact experience. Users are terrified, usually
for no good reason at all. I'll make sure that this comes across in
the next revision of the patch series.
> Also, we should be more clear about how it's different from VACUUMs triggered due to the scale factors (cancellation behavior, being triggered when autovacuum is disabled etc.).
Right. Though I think that the biggest point of confusion for users is
how *few* differences there really are between antiwraparound
autovacuum, and any other kind of autovacuum that happens to use
VACUUM's aggressive strategy. There is really only one important
difference: the autocancellation behavior. This is an autovacuum
behavior, not a VACUUM behavior -- so the "VACUUM side" doesn't know
anything about that at all.
> 5) Can we use a better name for the XidStopLimit mode? It seems like a very implementation centric name. Maybe a better version of "Running out of the XID space" or something like that?
Coming up with a new user-facing name for xidStopLimit is already on
my TODO list (it's surprisingly hard). I have used that name so far
because it unambiguously refers to the exact thing that I want to talk
about when discussing the worst case. Other than that, it's a terrible
name.
> 6) In the XidStopLimit mode section, it would be good to explain briefly why you could get to this scenario. It's not something which should happen in a normal running system unless you have a long running transaction or inactive replication slots or a badly configured system or something of that sort.
I agree that that's important. Note that there is already something
about "removable cutoff" being held back at the start of the
discussion of freezing -- that will prevent freezing in exactly the
same way as it prevents cleanup of dead tuples.
That will become a WARNING box in the next revision. There should also
be a similar, analogous WARNING box (about "removable cutoff" being
held back) much earlier on in the docs -- this should appear in
"25.2.1. Recovering Disk Space". Obviously this structure suggests
that there is an isomorphism between freezing and removing bloat. For
example, if you cannot "freeze" an XID that appears in some tuple's
xmax, then you also cannot remove that tuple because VACUUM only sees
it as a recently dead tuple (if xmax is >= OldestXmin/removable
cutoff, and from a deleter that already committed).
I don't think that we need to spell the "isomorphism" point out to the
reader directly, but having a subtle cue that that's how it works
seems like a good idea.
> If you got to this point, other than running VACUUM to get out of the situation, it's also important to figure out what got you there in the first place as many VACUUMs should have attempted to advance the relfrozenxid and failed.
It's also true that problems that can lead to the system entering
xidStopLimit mode aren't limited to cases where doing required
freezing is fundamentally impossible due to something holding back
"removable cutoff". It's also possible that VACUUM simply can't keep
up (though the failsafe has helped with that problem a lot).
I tend to agree that there needs to be more about this in the
xidStopLimit subsection (discussion of freezing being held back by
"removable cutoff" is insufficient), but FWIW that seems like it
should probably be treated as out of scope for this patch. It is more
the responsibility of the other patch [1] that aims to put the
xidStopLimit documentation on a better footing (and remove that
terrible HINT about single user mode).
Of course, that other patch is closely related to this patch -- the
precise boundaries are unclear at this point. In any case I think that
this should happen, because I think that it's a good idea.
> There are a few other small things I noticed along the way but my goal was to look at the overall structure.
Thanks again! This is very helpful.
[1] https://www.postgresql.org/message-id/flat/CAJ7c6TM2D277U2wH8X78kg8pH3tdUqebV3_JCJqAkYQFHCFzeg%40mai...
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-03 22:48 ` Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
1 sibling, 2 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-03 22:48 UTC (permalink / raw)
To: samay sharma <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Wed, May 3, 2023 at 2:59 PM Peter Geoghegan <[email protected]> wrote:
> Coming up with a new user-facing name for xidStopLimit is already on
> my TODO list (it's surprisingly hard). I have used that name so far
> because it unambiguously refers to the exact thing that I want to talk
> about when discussing the worst case. Other than that, it's a terrible
> name.
What about "XID allocation overload"? The implication that I'm going
for here is that the system was misconfigured, or there was otherwise
some kind of imbalance between XID supply and demand. It also seems to
convey the true gravity of the situation -- it's *bad*, to be sure,
but in many environments it's a survivable condition.
One possible downside of this name is that it could suggest that all
that needs to happen is for autovacuum to catch up on vacuuming. In
reality the user *will* probably have to do more than just wait before
the system's ability to allocate new XIDs returns, because (in all
likelihood) autovacuum just won't be able to catch up unless and until
the user (say) drops a replication slot. Even still, the name seems to
work; it describes the conceptual model of the system accurately. Even
before the user drops the replication slot, autovacuum will at least
*try* to get the system back to being able to allocate new XIDs once
more.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-04 22:18 ` samay sharma <[email protected]>
2023-05-05 02:06 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-12 01:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
1 sibling, 2 replies; 51+ messages in thread
From: samay sharma @ 2023-05-04 22:18 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
On Wed, May 3, 2023 at 3:48 PM Peter Geoghegan <[email protected]> wrote:
> On Wed, May 3, 2023 at 2:59 PM Peter Geoghegan <[email protected]> wrote:
> > Coming up with a new user-facing name for xidStopLimit is already on
> > my TODO list (it's surprisingly hard). I have used that name so far
> > because it unambiguously refers to the exact thing that I want to talk
> > about when discussing the worst case. Other than that, it's a terrible
> > name.
>
> What about "XID allocation overload"? The implication that I'm going
> for here is that the system was misconfigured, or there was otherwise
> some kind of imbalance between XID supply and demand. It also seems to
> convey the true gravity of the situation -- it's *bad*, to be sure,
> but in many environments it's a survivable condition.
>
My concern with the term "overload" is similar to what you expressed below.
It indicates that the situation is due to extra load on the system (or due
to too many XIDs being allocated) and people might assume that the
situation will resolve itself if the load were to be reduced / removed.
However, it's due to that along with some misconfiguration or some other
thing holding back the "removable cutoff".
What do you think about the term "Exhaustion"? Maybe something like "XID
allocation exhaustion" or "Exhaustion of allocatable XIDs"? The term
indicates that we are running out of XIDs to allocate without necessarily
pointing towards a reason.
Regards,
Samay
>
> One possible downside of this name is that it could suggest that all
> that needs to happen is for autovacuum to catch up on vacuuming. In
> reality the user *will* probably have to do more than just wait before
> the system's ability to allocate new XIDs returns, because (in all
> likelihood) autovacuum just won't be able to catch up unless and until
> the user (say) drops a replication slot. Even still, the name seems to
> work; it describes the conceptual model of the system accurately. Even
> before the user drops the replication slot, autovacuum will at least
> *try* to get the system back to being able to allocate new XIDs once
> more.
>
> --
> Peter Geoghegan
>
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
@ 2023-05-05 02:06 ` Peter Geoghegan <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-05 02:06 UTC (permalink / raw)
To: samay sharma <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Thu, May 4, 2023 at 3:18 PM samay sharma <[email protected]> wrote:
> What do you think about the term "Exhaustion"?
I'm really not sure.
Attached is v3, which (as with v1 and v2) comes with a prebuilt html
"Routine Vacuuming", for the convenience of reviewers.
v3 does have some changes based on your feedback (and feedback from
John), but overall v3 can be thought of as v2 with lots and lots of
additional copy-editing -- though still not enough, I'm sure.
v3 does add some (still incomplete) introductory remarks about the
intended audience and goals for "Routine Vacuuming". But most of the
changes are to the way the docs describe freezing and aggressive
VACUUM, which continued to be my focus for v3.
--
Peter Geoghegan
Attachments:
[text/html] routine-vacuuming.html (62.1K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/2-routine-vacuuming.html)
download | inline:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>25.2. Autovacuum Maintenance Tasks</title><link rel="stylesheet" type="text/css" href="https://www.postgresql.org/media/css/docs-complete.css" /><link rev="made" href="[email protected]" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><link rel="prev" href="autovacuum.html" title="25.1. The Autovacuum Daemon" /><link rel="next" href="routine-reindex.html" title="25.3. Routine Reindexing" /></head><body id="docContent" class="container-fluid col-10"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">25.2. Autovacuum Maintenance Tasks</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="autovacuum.html" title="25.1. The Autovacuum Daemon">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="maintenance.html" title="Chapter 25. Routine Database Maintenance Tasks">Up</a></td><th width="60%" align="center">Chapter 25. Routine Database Maintenance Tasks</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 16devel Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="routine-reindex.html" title="25.3. Routine Reindexing">Next</a></td></tr></table><hr /></div><div class="sect1" id="ROUTINE-VACUUMING"><div class="titlepage"><div><div><h2 class="title" style="clear: both">25.2. Autovacuum Maintenance Tasks <a href="#ROUTINE-VACUUMING" class="id_link">#</a></h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="routine-vacuuming.html#VACUUM-FOR-SPACE-RECOVERY">25.2.1. Recovering Disk Space</a></span></dt><dt><span class="sect2"><a href="routine-vacuuming.html#FREEZING-XID-SPACE">25.2.2. Freezing to manage the transaction ID space</a></span></dt><dt><span class="sect2"><a href="routine-vacuuming.html#VACUUM-FOR-VISIBILITY-MAP">25.2.3. Updating the Visibility Map</a></span></dt><dt><span class="sect2"><a href="routine-vacuuming.html#VACUUM-TRUNCATE-XACT-STATUS">25.2.4. Truncating transaction status information</a></span></dt><dt><span class="sect2"><a href="routine-vacuuming.html#VACUUM-FOR-STATISTICS">25.2.5. Updating Planner Statistics</a></span></dt></dl></div><a id="id-1.6.12.11.2" class="indexterm"></a><p>
<span class="productname">PostgreSQL</span> databases require periodic
maintenance known as <em class="firstterm">vacuuming</em>, and require
periodic updates to the statistics used by the
<span class="productname">PostgreSQL</span> query planner. The <a class="link" href="sql-vacuum.html" title="VACUUM"><code class="command">VACUUM</code></a> and <a class="link" href="sql-analyze.html" title="ANALYZE"><code class="command">ANALYZE</code></a> commands
perform these maintenance tasks. The <em class="firstterm">autovacuum
daemon</em> automatically schedules execution of
maintenance, based on the requirements of the workload.
</p><p>
The autovacuum daemon has to process each table regularly for
several reasons:
</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem">To recover or reuse disk space occupied by updated or deleted
rows.</li><li class="listitem">To maintain the system's ability to allocated new
transaction IDs through freezing.</li><li class="listitem">To update the visibility map, which speeds
up <a class="link" href="indexes-index-only-scans.html" title="11.9. Index-Only Scans and Covering Indexes">index-only
scans</a>, and helps the next <code class="command">VACUUM</code>
operation avoid needlessly scanning already-frozen pages.</li><li class="listitem">To truncate obsolescent transaction status information,
when possible.</li><li class="listitem">To update data statistics used by the
<span class="productname">PostgreSQL</span> query planner.</li></ol></div><p>
Maintenance work within the scope of items 1, 2, 3, and 4 is
performed by the <code class="command">VACUUM</code> command internally. The
<code class="command">ANALYZE</code> command handles maintenance work within
the scope of item 5 (maintenance of planner statistics) internally.
</p><p>
Generally speaking, database administrators that are new to tuning
autovacuum should start by considering adjusting autovacuum's
scheduling. Autovacuum scheduling is controlled via threshold
settings. These settings determine when autovacuum should launch a
worker to run <code class="command">VACUUM</code> and/or
<code class="command">ANALYZE</code>; see the previous section, <a class="xref" href="autovacuum.html" title="25.1. The Autovacuum Daemon">Section 25.1</a>. This section provides additional
information about the design and goals of autovacuum,
<code class="command">VACUUM</code>, and <code class="command">ANALYZE</code>. The
intended audience is database administrators that wish to perform
more advanced tuning of autovacuum, with any of the following goals
in mind:
</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
Tuning <code class="command">VACUUM</code> to improve query response times.
</p></li><li class="listitem"><p>
Making sure that <code class="command">VACUUM</code>'s management of the
Transaction ID address space is operating normally.
</p></li><li class="listitem"><p>
Tuning <code class="command">VACUUM</code> for performance stability.
</p></li></ul></div><p>
With larger installations, tuning autovacuum usually won't be a
once-off task; it is best to approach tuning as an iterative,
applied process. FIXME Expand this to describe the intended
audience on goals in a fully worked out way.
</p><p>
Autovacuum creates a substantial amount of I/O traffic, which can
cause poor performance for other active sessions. There are
configuration parameters that you can adjust to reduce the
performance impact of background vacuuming. See the
autovacuum-specific cost delay settings described in <a class="xref" href="runtime-config-autovacuum.html" title="20.10. Automatic Vacuuming">Section 20.10</a>, and additional cost delay
settings described in <a class="xref" href="runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-VACUUM-COST" title="20.4.4. Cost-based Vacuum Delay">Section 20.4.4</a>.
</p><p>
Some database administrators will want to supplement the daemon's
activities with manually-managed <code class="command">VACUUM</code>
commands. Scripting tools like <span class="application">cron</span> and
<span class="application">Task Manager</span> can be of help with this.
It can be useful to perform off-hours <code class="command">VACUUM</code>
commands during periods where reduced load is expected. Almost all
of the contents of this section apply equally to manually-issued
<code class="command">VACUUM</code> and <code class="command">ANALYZE</code>
operations.
</p><div class="sect2" id="VACUUM-FOR-SPACE-RECOVERY"><div class="titlepage"><div><div><h3 class="title">25.2.1. Recovering Disk Space <a href="#VACUUM-FOR-SPACE-RECOVERY" class="id_link">#</a></h3></div></div></div><a id="id-1.6.12.11.10.2" class="indexterm"></a><p>
In <span class="productname">PostgreSQL</span>, an
<code class="command">UPDATE</code> or <code class="command">DELETE</code> of a row does not
immediately remove the old version of the row.
This approach is necessary to gain the benefits of multiversion
concurrency control (<acronym class="acronym">MVCC</acronym>, see <a class="xref" href="mvcc.html" title="Chapter 13. Concurrency Control">Chapter 13</a>): the row version
must not be deleted while it is still potentially visible to other
transactions. A deleted row version (whether from an
<code class="command">UPDATE</code> or <code class="command">DELETE</code>) will
usually cease to be of interest to any still-running transaction
shortly after the original deleting transaction commits.
</p><p>
The space dead tuples occupy must eventually be reclaimed for
reuse by new rows, to avoid unbounded growth of disk space
requirements. Reclaiming space from dead rows is
<code class="command">VACUUM</code>'s main responsibility.
</p><p>
The <a class="glossterm" href="glossary.html#GLOSSARY-XID"><em class="glossterm"><a class="glossterm" href="glossary.html#GLOSSARY-XID" title="Transaction ID">transaction ID number
(<acronym class="acronym">XID</acronym>)</a></em></a> based cutoff point that
<code class="command">VACUUM</code> uses to determine if a deleted tuple is
safe to physically remove is reported under <code class="literal">removable
cutoff</code> in the server log when autovacuum logging
(controlled by <a class="xref" href="runtime-config-logging.html#GUC-LOG-AUTOVACUUM-MIN-DURATION">log_autovacuum_min_duration</a>)
reports on a <code class="command">VACUUM</code> operation executed by
autovacuum. Tuples that are not yet safe to remove are counted as
<code class="literal">dead but not yet removable</code> tuples in the log
report. <code class="command">VACUUM</code> establishes its
<code class="literal">removable cutoff</code> once, at the start of the
operation. Any older <acronym class="acronym">MVCC</acronym> snapshot (or
transaction that allocates an XID) that's still running when the
cutoff is established may hold it back.
</p><div class="caution"><h3 class="title">Caution</h3><p>
It's important that no long-running transactions ever be allowed
to hold back every <code class="command">VACUUM</code> operation's cutoff
for an extended period. You may wish to add monitoring to alert
on this.
</p></div><div class="note"><h3 class="title">Note</h3><p>
<code class="command">VACUUM</code> can remove tuples inserted by aborted
transactions immediately
</p></div><p>
<code class="command">VACUUM</code> usually won't return space to the
operating system. There is one exception: space is returned to the
OS whenever a group of contiguous pages appears at the end of a
table. <code class="command">VACUUM</code> must acquire an <code class="literal">ACCESS
EXCLUSIVE</code> lock to perform relation truncation. You can
disable relation truncation by setting the table's
<code class="varname">vacuum_truncate</code> storage parameter to
<code class="literal">off</code>.
</p><div class="tip"><h3 class="title">Tip</h3><p>
If you have a table whose entire contents are deleted on a
periodic basis, consider doing it with <a class="link" href="sql-truncate.html" title="TRUNCATE"><code class="command">TRUNCATE</code></a> rather
than relying on <code class="command">VACUUM</code>.
<code class="command">TRUNCATE</code> removes the entire contents of the
table immediately, avoiding the need to set
<code class="structfield">xmax</code> to the deleting transaction's XID.
One disadvantage is that strict MVCC semantics are violated.
</p></div><div class="tip"><h3 class="title">Tip</h3><p>
<code class="command">VACUUM FULL</code> or <code class="command">CLUSTER</code> can
be useful when dealing with extreme amounts of dead tuples. It
can reclaim more disk space, but runs much more slowly. It
rewrites an entire new copy of the table and rebuilds all of the
table's indexes. As a result, <code class="command">VACUUM FULL</code> and
<code class="command">CLUSTER</code> typically have a much higher overhead
than <code class="command">VACUUM</code>. Generally, therefore,
administrators should avoid using <code class="command">VACUUM FULL</code>
except in the most extreme cases.
</p></div><div class="note"><h3 class="title">Note</h3><p>
Although <code class="command">VACUUM FULL</code> is technically an option
of the <code class="command">VACUUM</code> command, <code class="command">VACUUM
FULL</code> uses a completely different implementation.
<code class="command">VACUUM FULL</code> is essentially a variant of
<code class="command">CLUSTER</code>. (The name <code class="command">VACUUM
FULL</code> is historical; the original implementation was
somewhat closer to standard <code class="command">VACUUM</code>.)
</p></div><div class="warning"><h3 class="title">Warning</h3><p>
<code class="command">TRUNCATE</code>, <code class="command">VACUUM FULL</code>, and
<code class="command">CLUSTER</code> all require an <code class="literal">ACCESS
EXCLUSIVE</code> lock, which can be highly disruptive
(<code class="command">SELECT</code>, <code class="command">INSERT</code>,
<code class="command">UPDATE</code>, and <code class="command">DELETE</code> commands
won't be able to run at the same time).
</p></div><div class="warning"><h3 class="title">Warning</h3><p>
<code class="command">VACUUM FULL</code> and <code class="command">CLUSTER</code>
temporarily use extra disk space. The extra space required is
approximately equal to the size of the table, since the old
copies of the table and indexes can't be released until the new
ones are complete.
</p></div></div><div class="sect2" id="FREEZING-XID-SPACE"><div class="titlepage"><div><div><h3 class="title">25.2.2. Freezing to manage the transaction ID space <a href="#FREEZING-XID-SPACE" class="id_link">#</a></h3></div></div></div><a id="id-1.6.12.11.11.2" class="indexterm"></a><p>
<code class="command">VACUUM</code> often marks some of the pages that it
scans <span class="emphasis"><em>frozen</em></span>, indicating that all eligible
rows on the page were inserted by a transaction that committed
sufficiently far in the past that the effects of the inserting
transaction are certain to be visible to all current and future
transactions. The specific Transaction ID number
(<acronym class="acronym">XID</acronym>) stored in a frozen heap row's
<code class="structfield">xmin</code> field is no longer needed to
determine anything about the row's visibility. Furthermore, when
a row undergoing freezing happens to have an XID set in its
<code class="structfield">xmax</code> field (possibly an XID left behind
by an earlier <code class="command">SELECT FOR UPDATE</code> row locker),
the <code class="structfield">xmax</code> field's XID is usually also
removed.
</p><p>
Once frozen, heap pages are <span class="quote">“<span class="quote">self-contained</span>”</span>. Every
query can read all of the page's rows in a way that assumes that
the inserting transaction committed and is visible to its
<acronym class="acronym">MVCC</acronym> snapshot. No query will ever have to
consult external transaction status metadata to interpret the
page's contents, either. In particular,
<code class="filename">pg_xact</code> transaction XID commit/abort status
lookups won't take place during query execution.
</p><p>
Freezing is a <acronym class="acronym">WAL</acronym>-logged operation, so when
<code class="command">VACUUM</code> freezes a heap page, any copy of the
page located on a physical replication standby server will itself
be <span class="quote">“<span class="quote">frozen</span>”</span> shortly thereafter (when the relevant
<code class="literal">FREEZE_PAGE</code> <acronym class="acronym">WAL</acronym> record is
replayed on the standby). Queries that run on physical
replication standbys thereby avoid <code class="filename">pg_xact</code>
lookups when reading from frozen pages, in just the same way as
queries that run on the primary server
<a href="#ftn.id-1.6.12.11.11.5.7" class="footnote"><sup class="footnote" id="id-1.6.12.11.11.5.7">[15]</sup></a>.
</p><p>
It can be useful for <code class="command">VACUUM</code> to put off some of
the work of freezing, but <code class="command">VACUUM</code> cannot put off
freezing forever. Since on-disk storage of transaction IDs in
heap row headers uses a truncated 32-bit representation to save
space (rather than the full 64-bit representation), freezing plays
a crucial role in enabling <a class="link" href="routine-vacuuming.html#AGGRESSIVE-VACUUM" title="25.2.2.1. Aggressive VACUUM">management of the XID address
space</a> by <code class="command">VACUUM</code>. If, for whatever
reason, <code class="command">VACUUM</code> is unable to freeze older XIDs
on behalf of an application that continues to require new XID
allocations, the system will eventually
<a class="link" href="routine-vacuuming.html#XID-STOP-LIMIT" title="25.2.2.3. xidStopLimit mode">refuse to allocate new transaction IDs</a>.
The system generally only enters this state when autovacuum is
misconfigured.
</p><p>
<a class="xref" href="runtime-config-client.html#GUC-VACUUM-FREEZE-MIN-AGE">vacuum_freeze_min_age</a> controls when freezing
takes place. When <code class="command">VACUUM</code> scans a heap page
containing even one XID that has already attained an age exceeding
this value, the page is frozen.
</p><a id="id-1.6.12.11.11.8" class="indexterm"></a><p>
<em class="firstterm">MultiXact IDs</em> are used to support row
locking by multiple transactions. Since there is only limited
space in a tuple header to store lock information, that
information is encoded as a <span class="quote">“<span class="quote">multiple transaction
ID</span>”</span>, or MultiXact ID for short, whenever there is more
than one transaction concurrently locking a row. Information
about which transaction IDs are included in any particular
MultiXact ID is stored separately in
<code class="filename">pg_multixact</code>, and only the MultiXact ID
itself (a 32-bit unsigned integer) appears in the tuple's
<code class="structfield">xmax</code> field. This creates a dependency
on external transaction status information similar to the
dependency that ordinary unfrozen XIDs have on commit status
information stored in <code class="filename">pg_xact</code>.
<code class="command">VACUUM</code> must therefore occasionally remove
MultiXact IDs from tuples during freezing.
</p><p>
<a class="xref" href="runtime-config-client.html#GUC-VACUUM-MULTIXACT-FREEZE-MIN-AGE">vacuum_multixact_freeze_min_age</a> also
controls when freezing takes place. It is analogous to
<code class="varname">vacuum_freeze_min_age</code>, but <span class="quote">“<span class="quote">age</span>”</span>
is expressed in units of MultiXact ID.
Lowering <code class="varname">vacuum_multixact_freeze_min_age</code>
<span class="emphasis"><em>forces</em></span> <code class="command">VACUUM</code> to process
<code class="structfield">xmax</code> fields containing a MultiXact ID
in cases where it would otherwise opt to put off the work of
processing <code class="structfield">xmax</code> until the next
<code class="command">VACUUM</code> <a href="#ftn.id-1.6.12.11.11.10.10" class="footnote"><sup class="footnote" id="id-1.6.12.11.11.10.10">[16]</sup></a>. The setting generally doesn't significantly
influence the total number of pages <code class="command">VACUUM</code>
freezes, even in tables that contain relatively many MultiXact
IDs. This is because <code class="command">VACUUM</code> generally prefers
proactively processing for most individual
<code class="structfield">xmax</code> fields that contain a MultiXact ID
(eager proactive processing is typically cheaper).
</p><p>
Managing the added <acronym class="acronym">WAL</acronym> volume from freezing
over time is an important consideration for
<code class="command">VACUUM</code>. It is why <code class="command">VACUUM</code>
doesn't just freeze every eligible tuple at the earliest
opportunity: the <acronym class="acronym">WAL</acronym> written to freeze a page's
tuples <span class="quote">“<span class="quote">goes to waste</span>”</span> in cases where the resulting
frozen tuples are soon deleted or updated anyway. It's also why
<code class="command">VACUUM</code> <span class="emphasis"><em>will</em></span> freeze all
eligible tuples from a heap page once the decision to freeze at
least one tuple is taken: at that point the added cost to freeze
all eligible tuples eagerly (measured in <span class="quote">“<span class="quote">extra bytes of
<acronym class="acronym">WAL</acronym> written</span>”</span>) is far lower than the
probable cost of deferring freezing until a future
<code class="command">VACUUM</code> operation against the same table.
Furthermore, once the page is frozen it can generally be marked as
all-frozen in the visibility map right away.
</p><div class="note"><h3 class="title">Note</h3><p>
In <span class="productname">PostgreSQL</span> versions before 16,
<code class="command">VACUUM</code> triggered freezing at the level of
individual <code class="structfield">xmin</code> and
<code class="structfield">xmax</code> fields. Freezing only affected
the exact XIDs that had already attained an age of
<code class="varname">vacuum_freeze_min_age</code> or greater.
</p></div><p>
<code class="command">VACUUM</code> also triggers freezing of a page in
cases where it already proved necessary to write out a full page
image (<acronym class="acronym">FPI</acronym>) as part of a <acronym class="acronym">WAL</acronym>
record describing how dead tuples were removed <a href="#ftn.id-1.6.12.11.11.13.4" class="footnote"><sup class="footnote" id="id-1.6.12.11.11.13.4">[17]</sup></a> (see <a class="xref" href="wal-reliability.html" title="30.1. Reliability">Section 30.1</a> for background
information about how <acronym class="acronym">FPI</acronym>s provide torn page
protection). This <span class="quote">“<span class="quote">freeze on an <acronym class="acronym">FPI</acronym>
write</span>”</span> batching mechanism often avoids the need for some
future <code class="command">VACUUM</code> operation to write an additional
<acronym class="acronym">FPI</acronym> for the same page as part of a
<acronym class="acronym">WAL</acronym> record describing how live tuples were
frozen. In effect, <code class="command">VACUUM</code> writes slightly more
<acronym class="acronym">WAL</acronym> in the short term with the aim of
ultimately needing to write much less <acronym class="acronym">WAL</acronym> in
the long term.
</p><div class="tip"><h3 class="title">Tip</h3><p>
For tables which receive <code class="command">INSERT</code> operations,
but few or no <code class="command">UPDATE</code>/<code class="command">DELETE</code>
operations, it may be beneficial to selectively lower <a class="xref" href="sql-createtable.html#RELOPTION-AUTOVACUUM-FREEZE-MIN-AGE">autovacuum_freeze_min_age</a> for the table.
<code class="command">VACUUM</code> may thereby be able to freeze the
table's pages <span class="quote">“<span class="quote">eagerly</span>”</span> during earlier autovacuums
triggered by <a class="xref" href="runtime-config-autovacuum.html#GUC-AUTOVACUUM-VACUUM-INSERT-SCALE-FACTOR">autovacuum_vacuum_insert_scale_factor</a>.
</p></div><div class="caution"><h3 class="title">Caution</h3><p>
<code class="command">VACUUM</code> may not be able to freeze every tuple's
<code class="structfield">xmin</code> in relatively rare cases. The
criteria that determines basic eligibility for freezing is the
same as the one that determines if a deleted tuple can be
removed: the XID-based <code class="literal">removable cutoff</code> that
appears in the server log's autovacuum log reports (controlled by
<a class="xref" href="runtime-config-logging.html#GUC-LOG-AUTOVACUUM-MIN-DURATION">log_autovacuum_min_duration</a>).
</p><p>
In extreme cases, a long-running transaction can hold back every
<code class="command">VACUUM</code>'s removable cutoff for so long that the
system is forced to activate <a class="link" href="routine-vacuuming.html#XID-STOP-LIMIT" title="25.2.2.3. xidStopLimit mode"><code class="literal">xidStopLimit</code> mode
protections</a>.
</p></div><div class="sect3" id="AGGRESSIVE-VACUUM"><div class="titlepage"><div><div><h4 class="title">25.2.2.1. Aggressive <code class="command">VACUUM</code> <a href="#AGGRESSIVE-VACUUM" class="id_link">#</a></h4></div></div></div><a id="id-1.6.12.11.11.16.2" class="indexterm"></a><a id="id-1.6.12.11.11.16.3" class="indexterm"></a><p>
As noted already, freezing doesn't just allow queries to avoid
lookups of subsidiary transaction status information in
structures such as <code class="filename">pg_xact</code>. Freezing also
plays a crucial role in enabling management of the XID address
space by <code class="command">VACUUM</code>. <code class="command">VACUUM</code>
maintains information about the oldest unfrozen XID that remains
in the table when it uses its <em class="firstterm">aggressive strategy</em>.
</p><p>
Aggressive <code class="command">VACUUM</code> will update the table's
<code class="structname">pg_class</code>.<code class="structfield">relfrozenxid</code>
to the value that it determined to be the oldest remaining XID;
the table's <code class="structfield">relfrozenxid</code>
<span class="quote">“<span class="quote">advances</span>”</span> by a certain number of XIDs. Aggressive
<code class="command">VACUUM</code> may also need to update the
<code class="structfield">datfrozenxid</code> column of the database's
<code class="structname">pg_database</code> row in turn.
<code class="structfield">datfrozenxid</code> is a lower bound on the
unfrozen XIDs appearing in that database — it is just the
minimum of the per-table <code class="structfield">relfrozenxid</code>
values (the <code class="structfield">relfrozenxid</code> that has
attained the greatest age) within the database.
</p><p>
Aggressive <code class="command">VACUUM</code> also maintains the
<code class="structname">pg_class</code>.<code class="structfield">relminmxid</code>
and <code class="structname">pg_database</code>.<code class="structfield">datminmxid</code>
fields. These are needed to track the oldest MultiXact ID that
remains in the table and database, respectively.
</p><p>
The extra steps performed within every aggressive
<code class="command">VACUUM</code> against every table have the overall
effect of tracking the oldest remaining unfrozen transaction ID
in the entire cluster (every table from every database).
Aggressive <code class="command">VACUUM</code>s will (in the aggregate and
over time) make sure that the oldest unfrozen transaction ID in
the entire system is never too far in the past.
</p><div class="note"><h3 class="title">Managing the Transaction ID Space</h3><p>
Freezing removes <span class="emphasis"><em>local</em></span> dependencies on
external transaction status information from individual heap
pages. Advancing <code class="structfield">relfrozenxid</code>
removes <span class="emphasis"><em>global</em></span> dependencies from whole
tables in turn.
</p><p>
The oldest XID in the entire cluster can be thought of as the
beginning of the XID space, while the next unallocated XID can
be thought of as the end of the XID space. This space
represents the range of XIDs that might still require
transaction commit/abort status lookups in <code class="filename">pg_xact</code>.
</p></div><p>
The maximum XID age that the system can tolerate (i.e., the
maximum <span class="quote">“<span class="quote">distance</span>”</span> between the oldest unfrozen
transaction ID in any table according to
<code class="structname">pg_class</code>.<code class="structfield">relfrozenxid</code>,
and the next unallocated transaction ID) is about 2.1 billion
transaction IDs. This <span class="quote">“<span class="quote">maximum XID age</span>”</span> invariant
makes it fundamentally impossible to put off aggressive
<code class="command">VACUUM</code>s (and freezing) forever
<a href="#ftn.id-1.6.12.11.11.16.9.6" class="footnote"><sup class="footnote" id="id-1.6.12.11.11.16.9.6">[18]</sup></a>. The invariant imposes an absolute hard limit on how
long any table can go without an aggressive <code class="command">VACUUM</code>.
</p><p>
If the hard limit is ever reached, then the system will activate
<a class="link" href="routine-vacuuming.html#XID-STOP-LIMIT" title="25.2.2.3. xidStopLimit mode"><code class="literal">xidStopLimit</code>
mode</a>, which temporarily prevents the allocation of new
permanent transaction IDs. The system will only deactive
<code class="literal">xidStopLimit</code> mode when
<code class="command">VACUUM</code> (typically run by autovacuum) succeeds
in advancing the oldest <code class="literal">datfrozenxid</code> in the
cluster (via an aggressive <code class="command">VACUUM</code> that runs to
completion against the table that has the oldest
<code class="structfield">relfrozenxid</code>).
</p><p>
The 2.1 billion XIDs <span class="quote">“<span class="quote">maximum XID age</span>”</span> invariant
must be preserved because transaction IDs stored in heap row
headers use a truncated 32-bit representation (rather than the
full 64-bit representation). Since all unfrozen transaction IDs
from heap tuple headers <span class="emphasis"><em>must</em></span> be from the
same transaction ID epoch (or from a space in the 64-bit
representation that spans two adjoining transaction ID epochs),
there isn't any need to store a separate epoch field in each
tuple header (see <a class="xref" href="transaction-id.html#INTERPRETING-XID-STAMPS" title="74.1.2.1. TransactionId comparison rules">Section 74.1.2.1</a> for
further details). This scheme has the advantage of requiring
much less on-disk storage space than a design that stores an XID
epoch alongside each XID stored in each heap tuple header. It
has the disadvantage of constraining the system's ability to
allocate new XIDs in the worst case scenario where
<code class="literal">xidStopLimit</code> mode is used to preserve the
<span class="quote">“<span class="quote">maximum XID age</span>”</span> invariant.
</p><p>
There is only one major runtime behavioral difference between
aggressive mode <code class="command">VACUUM</code>s and non-aggressive
<code class="command">VACUUM</code>s: only non-aggressive
<code class="command">VACUUM</code>s will skip pages that don't have any
dead row versions even if those pages still have row versions
with old XID values (pages marked as all-visible in the
visibility map). Aggressive <code class="command">VACUUM</code>s can only
skip pages that are marked as both all-visible and all-frozen.
Consequently, non-aggressive <code class="command">VACUUM</code>s usually
won't freeze <span class="emphasis"><em>every</em></span> page containing an XID
that has already attained an age of
<code class="varname">vacuum_freeze_min_age</code> or more. Failing to
freeze older pages during non-aggressive
<code class="command">VACUUM</code>s may lead to aggressive
<code class="command">VACUUM</code>s that perform a disproportionately
large amount of the work of freezing required by one particular
table.
</p><div class="tip"><h3 class="title">Tip</h3><p>
When the <code class="command">VACUUM</code> command's
<code class="literal">VERBOSE</code> parameter is specified,
<code class="command">VACUUM</code> prints various statistics about the
table. Its output includes information about how
<code class="structfield">relfrozenxid</code> and
<code class="structfield">relminmxid</code> advanced, and the number
of newly frozen pages. The same details appear in the server
log when autovacuum logging (controlled by <a class="xref" href="runtime-config-logging.html#GUC-LOG-AUTOVACUUM-MIN-DURATION">log_autovacuum_min_duration</a>) reports on a
<code class="command">VACUUM</code> operation executed by autovacuum.
</p></div><div class="note"><h3 class="title">Note</h3><p>
In practice, most tables require periodic aggressive vacuuming.
However, some individual non-aggressive
<code class="command">VACUUM</code> operations may be able to advance
<code class="structfield">relfrozenxid</code> and/or
<code class="structfield">relminmxid</code>. Non-aggressive
<code class="structfield">relfrozenxid</code>/<code class="structfield">relminmxid</code>
advancement is most common in small, frequently modified tables.
</p></div><p>
Most individual tables will eventually need an aggressive
<code class="command">VACUUM</code>, which will reliably freeze all pages
with XID (or MultiXact ID) values older than
<code class="varname">vacuum_freeze_min_age</code> (or older than
<code class="varname">vacuum_multixact_freeze_min_age</code>), including
those from all-visible but not all-frozen pages (and then advance
<code class="structname">pg_class</code>.<code class="structfield">relfrozenxid</code>
to a value that reflects all that). <a class="xref" href="runtime-config-client.html#GUC-VACUUM-FREEZE-TABLE-AGE">vacuum_freeze_table_age</a> controls when
<code class="command">VACUUM</code> must use its aggressive strategy. If
<code class="literal">age(relfrozenxid)</code> exceeds
<code class="varname">vacuum_freeze_table_age</code> at the start of
<code class="command">VACUUM</code>, that <code class="command">VACUUM</code> will
use the aggressive strategy; otherwise the standard
non-aggressive strategy is used. Setting
<code class="varname">vacuum_freeze_table_age</code> to 0 forces
<code class="command">VACUUM</code> to always use its aggressive strategy.
</p></div><div class="sect3" id="ANTI-WRAPAROUND-AUTOVACUUMS"><div class="titlepage"><div><div><h4 class="title">25.2.2.2. Anti-Wraparound Autovacuums <a href="#ANTI-WRAPAROUND-AUTOVACUUMS" class="id_link">#</a></h4></div></div></div><p>
To ensure that every table has its
<code class="structfield">relfrozenxid</code> advanced at somewhat
regular intervals, even in the case of completely static tables,
autovacuum runs against any table that might contain unfrozen
rows with XIDs older than the age specified by the configuration
parameter <a class="xref" href="runtime-config-autovacuum.html#GUC-AUTOVACUUM-FREEZE-MAX-AGE">autovacuum_freeze_max_age</a>. These
are <em class="firstterm">anti-wraparound autovacuums</em>.
Anti-wraparound autovacuums can happen even when autovacuum is
nominally disabled in <code class="filename">postgresql.conf</code>.
</p><p>
In practice, all anti-wraparound autovacuums will use
<code class="command">VACUUM</code>'s aggressive strategy (if they didn't,
then it would defeat the whole purpose of anti-wraparound
autovacuuming). Use of <code class="command">VACUUM</code>'s aggressive
strategy is certain, because the effective value of
<code class="varname">vacuum_freeze_table_age</code> is silently
<span class="quote">“<span class="quote">clamped</span>”</span> to a value no greater than 95% of the
current value of <code class="varname">autovacuum_freeze_max_age</code>.
</p><p>
As a rule of thumb, <code class="command">vacuum_freeze_table_age</code>
should be set to a value somewhat below
<code class="varname">autovacuum_freeze_max_age</code>, so that there is a
window during which any autovacuum triggered by inserts, updates,
or deletes (or any manually issued <code class="command">VACUUM</code>)
will become an aggressive <code class="command">VACUUM</code>. Such
<code class="command">VACUUM</code>s will reliably advance
<code class="structfield">relfrozenxid</code> in passing, even though
autovacuum won't have specifically set out to make sure
<code class="structfield">relfrozenxid</code> advances through
anti-wraparound autovacuuming. Anti-wraparound autovacuums may
never be required at all in tables that regularly require
vacuuming to <a class="link" href="routine-vacuuming.html#VACUUM-FOR-SPACE-RECOVERY" title="25.2.1. Recovering Disk Space">reclaim
space from dead tuples</a> and/or to <a class="link" href="routine-vacuuming.html#VACUUM-FOR-VISIBILITY-MAP" title="25.2.3. Updating the Visibility Map">set pages all-visible in the
visibility map</a> (especially if
<code class="varname">vacuum_freeze_table_age</code> is set to a value
significantly below
<code class="varname">autovacuum_freeze_max_age</code>).
</p><div class="note"><h3 class="title">Note on terminology</h3><p>
Aggressive <code class="command">VACUUM</code> is a special form of
<code class="command">VACUUM</code>. An aggressive
<code class="command">VACUUM</code> must advance
<code class="structfield">relfrozenxid</code> up to an XID value that
is no greater than <code class="varname">vacuum_freeze_min_age</code> XIDs
in age as of the <span class="emphasis"><em>start</em></span> of the
<code class="command">VACUUM</code> operation.
</p><p>
Anti-wraparound autovacuum is a special form of Autovacuum. Its
purpose is to make sure that
<code class="structfield">relfrozenxid</code> is advanced when no
earlier aggressive <code class="command">VACUUM</code> ran and advanced
<code class="structfield">relfrozenxid</code> in passing (often
because no <code class="command">VACUUM</code> needed to run against the
table at all).
</p><p>
There is only one runtime behavioral difference between
anti-wraparound autovacuums and other autovacuums that happen to
end up running an aggressive <code class="command">VACUUM</code>:
Anti-wraparound autovacuums <span class="emphasis"><em>cannot be
autocancelled</em></span>. This means that autovacuum workers
that perform anti-wraparound autovacuuming do not yield to
conflicting relation-level lock requests (e.g., from
<code class="command">ALTER TABLE</code>). See <a class="xref" href="autovacuum.html#AUTOVACUUM-LOCK-CONFLICTS" title="25.1.3. Autovacuum and Lock Conflicts">Section 25.1.3</a> for a full explanation.
</p></div><p>
<code class="command">VACUUM</code> also applies <a class="xref" href="runtime-config-client.html#GUC-VACUUM-MULTIXACT-FREEZE-TABLE-AGE">vacuum_multixact_freeze_table_age</a> and <a class="xref" href="runtime-config-autovacuum.html#GUC-AUTOVACUUM-MULTIXACT-FREEZE-MAX-AGE">autovacuum_multixact_freeze_max_age</a>. These are
independent MultiXact ID based triggers of aggressive
<code class="command">VACUUM</code> (and anti-wraparound autovacuum). They
are applied by following rules analogous to the rules already
described for <code class="varname">vacuum_freeze_table_age</code> and
<code class="varname">autovacuum_freeze_max_age</code>, respectively
<a href="#ftn.id-1.6.12.11.11.17.6.7" class="footnote"><sup class="footnote" id="id-1.6.12.11.11.17.6.7">[19]</sup></a>.
</p><p>
It doesn't matter if it was <code class="varname">vacuum_freeze_table_age</code> or
<code class="varname">vacuum_multixact_freeze_table_age</code> that
triggered <code class="command">VACUUM</code>'s decision to use its
aggressive strategy. <span class="emphasis"><em>Every</em></span> aggressive
<code class="command">VACUUM</code> will advance
<code class="structfield">relfrozenxid</code> and
<code class="structfield">relminmxid</code> by following the same
generic steps at runtime.
</p><p>
A convenient way to examine information about
<code class="structfield">relfrozenxid</code> and
<code class="structfield">relminmxid</code> is to execute queries such as:
</p><pre class="programlisting">
SELECT c.oid::regclass as table_name,
greatest(age(c.relfrozenxid),
age(t.relfrozenxid)) as xid_age,
mxid_age(c.relminmxid)
FROM pg_class c
LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm');
SELECT datname,
age(datfrozenxid) as xid_age,
mxid_age(datminmxid)
FROM pg_database;
</pre><p>
The <code class="function">age</code> function returns the number of
transactions from <code class="structfield">relfrozenxid</code> to the
next unallocated transaction ID. The
<code class="function">mxid_age</code> function the number of MultiXact
IDs from <code class="structfield">relminmxid</code> to the next
unallocated MultiXact ID.
</p><p>
The system should always have significant XID allocation slack
capacity. Ideally, the greatest
<code class="literal">age(relfrozenxid)</code>/<code class="literal">age(datfrozenxid)</code>
in the system will never be more than a fraction of the 2.1
billion XID hard limit described in <a class="xref" href="routine-vacuuming.html#AGGRESSIVE-VACUUM" title="25.2.2.1. Aggressive VACUUM">Section 25.2.2.1</a>. The default
<code class="varname">vacuum_freeze_table_age</code> setting of 200 million
transactions implies that the system should never use
significantly more than about 10% of that hard limit.
</p><p>
There is little advantage in routinely allowing the greatest
<code class="literal">age(relfrozenxid)</code> in the system to get
anywhere near to the 2.1 billion XID hard limit. Putting off the
work of freezing can only reduce the absolute amount of
<acronym class="acronym">WAL</acronym> written by <code class="command">VACUUM</code> when
<code class="command">VACUUM</code> thereby completely avoids freezing rows
that are deleted before long anyway. There is little or no
disadvantage from lowering <code class="varname">vacuum_freeze_table_age</code>
to make aggressive <code class="command">VACUUM</code>s more frequent, at
least in tables where newly frozen pages almost always remain
all-frozen forever. Note also that anything that leads to
<code class="structfield">relfrozenxid</code> and
<code class="structfield">relminmxid</code> advancing less frequently
(such as a higher <code class="varname">vacuum_freeze_table_age</code>
setting) will also increase the on-disk space required to store
additional transaction status information, as described in <a class="xref" href="routine-vacuuming.html#VACUUM-TRUNCATE-XACT-STATUS" title="25.2.4. Truncating transaction status information">Section 25.2.4</a>.
</p></div><div class="sect3" id="XID-STOP-LIMIT"><div class="titlepage"><div><div><h4 class="title">25.2.2.3. <code class="literal">xidStopLimit</code> mode <a href="#XID-STOP-LIMIT" class="id_link">#</a></h4></div></div></div><p>
If for some reason autovacuum utterly fails to advance any
table's <code class="structfield">relfrozenxid</code> or
<code class="structfield">relminmxid</code> for an extended period, and
if XIDs and/or MultiXact IDs continue to be allocated, the system
will begin to emit warning messages like this when the database's
oldest XIDs reach forty million transactions from the 2.1 billion
XID hard limit described in <a class="xref" href="routine-vacuuming.html#AGGRESSIVE-VACUUM" title="25.2.2.1. Aggressive VACUUM">Section 25.2.2.1</a>:
</p><pre class="programlisting">
WARNING: database "mydb" must be vacuumed within 39985967 transactions
HINT: To avoid a database shutdown, execute a database-wide VACUUM in that database.
</pre><p>
(A manual <code class="command">VACUUM</code> should fix the problem, as suggested by the
hint; but note that the <code class="command">VACUUM</code> must be performed by a
superuser, else it will fail to process system catalogs and thus not
be able to advance the database's <code class="structfield">datfrozenxid</code>.)
If these warnings are ignored, the system will eventually refuse
to start any new transactions. This happens at the point that
there are fewer than three million transactions left:
</p><pre class="programlisting">
ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb"
HINT: Stop the postmaster and vacuum that database in single-user mode.
</pre><p>
The three-million-transaction safety margin exists to let the
administrator recover without data loss, by manually executing the
required <code class="command">VACUUM</code> commands. However, since the system will not
execute commands once it has gone into the safety shutdown mode,
the only way to do this is to stop the server and start the server in single-user
mode to execute <code class="command">VACUUM</code>. The shutdown mode is not enforced
in single-user mode. See the <a class="xref" href="app-postgres.html" title="postgres"><span class="refentrytitle"><span class="application">postgres</span></span></a> reference
page for details about using single-user mode.
</p><p>
In emergencies, <code class="command">VACUUM</code> will take extraordinary
measures to avoid <code class="literal">xidStopLimit</code> mode. A
failsafe mechanism is triggered when the table's
<code class="structfield">relfrozenxid</code> attains an age of <a class="xref" href="runtime-config-client.html#GUC-VACUUM-FAILSAFE-AGE">vacuum_failsafe_age</a> XIDs, or when the table's
<code class="structfield">relminmxid</code> attains an age of <a class="xref" href="runtime-config-client.html#GUC-VACUUM-MULTIXACT-FAILSAFE-AGE">vacuum_multixact_failsafe_age</a> MultiXact IDs.
The failsafe prioritizes advancing
<code class="structfield">relfrozenxid</code> and/or
<code class="structfield">relminmxid</code> as quickly as possible.
Once the failsafe triggers, <code class="command">VACUUM</code> bypasses
all remaining non-essential maintenance tasks, and stops applying
any cost-based delay that was in effect. Any <a class="glossterm" href="glossary.html#GLOSSARY-BUFFER-ACCESS-STRATEGY"><em class="glossterm"><a class="glossterm" href="glossary.html#GLOSSARY-BUFFER-ACCESS-STRATEGY" title="Buffer Access Strategy">Buffer Access
Strategy</a></em></a> in use will also be disabled.
</p></div></div><div class="sect2" id="VACUUM-FOR-VISIBILITY-MAP"><div class="titlepage"><div><div><h3 class="title">25.2.3. Updating the Visibility Map <a href="#VACUUM-FOR-VISIBILITY-MAP" class="id_link">#</a></h3></div></div></div><p>
<code class="command">VACUUM</code> maintains a <a class="link" href="storage-vm.html" title="73.4. Visibility Map">visibility map</a> for each table to keep
track of which pages contain only tuples that are known to be
visible to all active transactions (and all future transactions,
at least until the page is modified). A separate bit tracks
whether all of the tuples are frozen.
</p><p>
The visibility map serves two purposes.
</p><p>
First, <code class="command">VACUUM</code> itself can skip such pages on the
next run, since there is nothing to clean up. Even <a class="link" href="routine-vacuuming.html#AGGRESSIVE-VACUUM" title="25.2.2.1. Aggressive VACUUM">aggressive <code class="command">VACUUM</code>s</a>
can skip pages that are both all-visible and all-frozen.
</p><p>
Second, it allows <span class="productname">PostgreSQL</span> to answer
some queries using only the index, without reference to the
underlying table. Since <span class="productname">PostgreSQL</span>
indexes don't contain tuple visibility information, a normal index
scan fetches the heap tuple for each matching index entry, to
check whether it should be seen by the current transaction. An
<a class="link" href="indexes-index-only-scans.html" title="11.9. Index-Only Scans and Covering Indexes"><em class="firstterm">index-only
scan</em></a>, on the other hand, checks the
visibility map first. If it's known that all tuples on the page
are visible, the heap fetch can be skipped. This is most useful
on large data sets where the visibility map can prevent disk
accesses. The visibility map is vastly smaller than the heap, so
it can easily be cached even when the heap is very large.
</p></div><div class="sect2" id="VACUUM-TRUNCATE-XACT-STATUS"><div class="titlepage"><div><div><h3 class="title">25.2.4. Truncating transaction status information <a href="#VACUUM-TRUNCATE-XACT-STATUS" class="id_link">#</a></h3></div></div></div><p>
Anything that influences when and how
<code class="structfield">relfrozenxid</code> and
<code class="structfield">relminmxid</code> advance will also directly
affect the high watermark storage overhead needed to store
historical transaction status information. For example,
increasing <code class="varname">autovacuum_freeze_max_age</code> (and
<code class="varname">vacuum_freeze_table_age</code> along with it) will
make the <code class="filename">pg_xact</code> and
<code class="filename">pg_commit_ts</code> subdirectories of the database
cluster take more space, because they store the commit/abort
status and (if <code class="varname">track_commit_timestamp</code> is enabled)
timestamp of all transactions back to the
<code class="varname">datfrozenxid</code> horizon (the earliest
<code class="varname">datfrozenxid</code> among all databases in the
cluster).
</p><p>
The commit status uses two bits per transaction. The default
<code class="varname">autovacuum_freeze_max_age</code> setting of 200
million transactions translates to about 50MB of
<code class="filename">pg_xact</code> storage. When
<code class="varname">track_commit_timestamp</code> is enabled, about 2GB of
<code class="filename">pg_commit_ts</code> storage will also be required.
</p><p>
MultiXact ID status information storage uses two separate
underlying <acronym class="acronym">SLRU</acronym> storage areas:
<code class="filename">pg_multixact/members</code>, and
<code class="filename">pg_multixact/offsets</code>. There is no simple
formula to determine the storage overhead per MultiXact ID, since
in general MultiXact IDs have a variable number of member XIDs.
Note, however, that if <code class="filename">pg_multixact/members</code>
exceeds 2GB, then the effective value of
<code class="varname">autovacuum_multixact_freeze_max_age</code> used by
<code class="command">VACUUM</code> will be lower, resulting in more
frequent aggressive mode <code class="command">VACUUM</code>s.
</p><p>
Truncation of transaction status information is only possible at
the end of <code class="command">VACUUM</code>s that advance the earliest
<code class="structfield">relfrozenxid</code> (in the case of
<code class="filename">pg_xact</code> and
<code class="filename">pg_commit_ts</code>), or the earliest
<code class="structfield">relminmxid</code> (in the case of
<code class="filename">pg_multixact/members</code> and
<code class="filename">pg_multixact/offsets</code>) among all tables in the
entire database (assuming that its the database with the earliest
<code class="structfield">datfrozenxid</code> and
<code class="structfield">datminmxid</code> in the entire cluster).
</p></div><div class="sect2" id="VACUUM-FOR-STATISTICS"><div class="titlepage"><div><div><h3 class="title">25.2.5. Updating Planner Statistics <a href="#VACUUM-FOR-STATISTICS" class="id_link">#</a></h3></div></div></div><a id="id-1.6.12.11.14.2" class="indexterm"></a><a id="id-1.6.12.11.14.3" class="indexterm"></a><p>
The <span class="productname">PostgreSQL</span> query planner relies on
statistical information about the contents of tables in order to
generate good plans for queries. These statistics are gathered by
the <a class="link" href="sql-analyze.html" title="ANALYZE"><code class="command">ANALYZE</code></a> command,
which can be invoked by itself or
as an optional step in <code class="command">VACUUM</code>. It is important to have
reasonably accurate statistics, otherwise poor choices of plans might
degrade database performance.
</p><p>
The autovacuum daemon, if enabled, will automatically issue
<code class="command">ANALYZE</code> commands whenever the content of a table has
changed sufficiently. However, administrators might prefer to rely
on manually-scheduled <code class="command">ANALYZE</code> operations, particularly
if it is known that update activity on a table will not affect the
statistics of <span class="quote">“<span class="quote">interesting</span>”</span> columns. The daemon schedules
<code class="command">ANALYZE</code> strictly as a function of the number of rows
inserted or updated; it has no knowledge of whether that will lead
to meaningful statistical changes.
</p><p>
Tuples changed in partitions and inheritance children do not trigger
analyze on the parent table. If the parent table is empty or rarely
changed, it may never be processed by autovacuum, and the statistics for
the inheritance tree as a whole won't be collected. It is necessary to
run <code class="command">ANALYZE</code> on the parent table manually in order to
keep the statistics up to date.
</p><p>
As with vacuuming for space recovery, frequent updates of statistics
are more useful for heavily-updated tables than for seldom-updated
ones. But even for a heavily-updated table, there might be no need for
statistics updates if the statistical distribution of the data is
not changing much. A simple rule of thumb is to think about how much
the minimum and maximum values of the columns in the table change.
For example, a <code class="type">timestamp</code> column that contains the time
of row update will have a constantly-increasing maximum value as
rows are added and updated; such a column will probably need more
frequent statistics updates than, say, a column containing URLs for
pages accessed on a website. The URL column might receive changes just
as often, but the statistical distribution of its values probably
changes relatively slowly.
</p><p>
It is possible to run <code class="command">ANALYZE</code> on specific tables and even
just specific columns of a table, so the flexibility exists to update some
statistics more frequently than others if your application requires it.
In practice, however, it is usually best to just analyze the entire
database, because it is a fast operation. <code class="command">ANALYZE</code> uses a
statistically random sampling of the rows of a table rather than reading
every single row.
</p><div class="tip"><h3 class="title">Tip</h3><p>
Although per-column tweaking of <code class="command">ANALYZE</code> frequency might not be
very productive, you might find it worthwhile to do per-column
adjustment of the level of detail of the statistics collected by
<code class="command">ANALYZE</code>. Columns that are heavily used in <code class="literal">WHERE</code>
clauses and have highly irregular data distributions might require a
finer-grain data histogram than other columns. See <code class="command">ALTER TABLE
SET STATISTICS</code>, or change the database-wide default using the <a class="xref" href="runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET">default_statistics_target</a> configuration parameter.
</p><p>
Also, by default there is limited information available about
the selectivity of functions. However, if you create a statistics
object or an expression
index that uses a function call, useful statistics will be
gathered about the function, which can greatly improve query
plans that use the expression index.
</p></div><div class="tip"><h3 class="title">Tip</h3><p>
The autovacuum daemon does not issue <code class="command">ANALYZE</code> commands for
foreign tables, since it has no means of determining how often that
might be useful. If your queries require statistics on foreign tables
for proper planning, it's a good idea to run manually-managed
<code class="command">ANALYZE</code> commands on those tables on a suitable schedule.
</p></div><div class="tip"><h3 class="title">Tip</h3><p>
The autovacuum daemon does not issue <code class="command">ANALYZE</code> commands
for partitioned tables. Inheritance parents will only be analyzed if the
parent itself is changed - changes to child tables do not trigger
autoanalyze on the parent table. If your queries require statistics on
parent tables for proper planning, it is necessary to periodically run
a manual <code class="command">ANALYZE</code> on those tables to keep the statistics
up to date.
</p></div></div><div class="footnotes"><br /><hr style="width:100; text-align:left;margin-left: 0" /><div id="ftn.id-1.6.12.11.11.5.7" class="footnote"><p><a href="#id-1.6.12.11.11.5.7" class="para"><sup class="para">[15] </sup></a>
In this regard freezing is unlike setting transaction status
<span class="quote">“<span class="quote">hint bits</span>”</span> in tuple headers: setting hint bits
doesn't usually need to be <acronym class="acronym">WAL</acronym>-logged, and
can take place on physical replication standby servers without
the involvement of the primary server. The purpose of hint bits
is to avoid repeat <code class="filename">pg_xact</code> lookups for the
same tuples, strictly as an optimization. The purpose of
freezing (from the point of view of individual tuples) is to
<span class="emphasis"><em>reliably</em></span> remove each tuple's dependency on
<code class="filename">pg_xact</code>, ultimately making it safe to
truncate <code class="filename">pg_xact</code> from time to time.
</p></div><div id="ftn.id-1.6.12.11.11.10.10" class="footnote"><p><a href="#id-1.6.12.11.11.10.10" class="para"><sup class="para">[16] </sup></a>
<span class="quote">“<span class="quote">Freezing</span>”</span> of <code class="structfield">xmax</code>
fields (whether they were found to contain an XID or a MultiXact
ID) generally means clearing <code class="structfield">xmax</code>.
<code class="command">VACUUM</code> may occasionally encounter an
individual MultiXact ID that must be removed to advance
<code class="structfield">relminmxid</code> by the required amount,
which can only be processed by generating a replacement
MultiXact ID (containing just the non-removable subset of member
XIDs from the original MultiXact ID), and then setting the
tuple's <code class="structfield">xmax</code> to the new/replacement
MultiXact ID value.
</p></div><div id="ftn.id-1.6.12.11.11.13.4" class="footnote"><p><a href="#id-1.6.12.11.11.13.4" class="para"><sup class="para">[17] </sup></a>
Actually, the <span class="quote">“<span class="quote">freeze on an <acronym class="acronym">FPI</acronym>
write</span>”</span> mechanism isn't just triggered whenever
<code class="command">VACUUM</code> needed to write an
<acronym class="acronym">FPI</acronym> for torn page protection as part of
writing a <code class="literal">PRUNE</code> <acronym class="acronym">WAL</acronym> record
describing how dead tuples were removed. The
<acronym class="acronym">FPI</acronym> mechanism can also be triggered when hint
bits are set by <code class="command">VACUUM</code>, if and only if doing
so necessitates writing an <acronym class="acronym">FPI</acronym>.
<acronym class="acronym">WAL</acronym>-logging in order to set hint bits is only
possible when the <a class="xref" href="runtime-config-wal.html#GUC-WAL-LOG-HINTS">wal_log_hints</a> option is
enabled in <code class="filename">postgresql.conf</code>, or when data
checksums were enabled when the cluster was initialized with
<a class="xref" href="app-initdb.html" title="initdb"><span class="refentrytitle"><span class="application">initdb</span></span></a>.
</p></div><div id="ftn.id-1.6.12.11.11.16.9.6" class="footnote"><p><a href="#id-1.6.12.11.11.16.9.6" class="para"><sup class="para">[18] </sup></a>
Aggressive <code class="command">VACUUM</code>s cannot be put off
forever, <span class="emphasis"><em>barring the edge-case where the
installation is never expected to consume more than about 2.1
billion XIDs</em></span>. In practice this has practical
relevance.
</p></div><div id="ftn.id-1.6.12.11.11.17.6.7" class="footnote"><p><a href="#id-1.6.12.11.11.17.6.7" class="para"><sup class="para">[19] </sup></a>
Though note that autovacuum (and <code class="command">VACUUM</code>) use a lower
<span class="quote">“<span class="quote">effective</span>”</span>
<code class="varname">autovacuum_multixact_freeze_max_age</code>
value (determined dynamically) to deal with issues with
truncation of the <acronym class="acronym">SLRU</acronym> storage areas, as
explained in <a class="xref" href="routine-vacuuming.html#VACUUM-TRUNCATE-XACT-STATUS" title="25.2.4. Truncating transaction status information">Section 25.2.4</a>
</p></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="autovacuum.html" title="25.1. The Autovacuum Daemon">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="maintenance.html" title="Chapter 25. Routine Database Maintenance Tasks">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="routine-reindex.html" title="25.3. Routine Reindexing">Next</a></td></tr><tr><td width="40%" align="left" valign="top">25.1. The Autovacuum Daemon </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 16devel Documentation">Home</a></td><td width="40%" align="right" valign="top"> 25.3. Routine Reindexing</td></tr></table></div></body></html>
[application/octet-stream] v3-0001-Make-autovacuum-docs-into-a-sect1-of-its-own.patch (18.2K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/3-v3-0001-Make-autovacuum-docs-into-a-sect1-of-its-own.patch)
download | inline diff:
From 0870911aab7bc3a7bff71e93d8d355b0e48640b7 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Wed, 12 Apr 2023 14:42:06 -0700
Subject: [PATCH v3 1/9] Make autovacuum docs into a sect1 of its own.
This doesn't change any of the content itself. Though it does move it
from the end of "Routine Vacuuming" (which is itself a sect1) to a whole
new sect1 that appears _before_ "Routine Vacuuming".
XXX Open question: does it make more sense to move the sect1 to before
"Routine Vacuuming", or should it go after instead? There are arguments
for both.
Arguments for "before":
"Before" gives greater prominence to the autovacuum scheduling tunables,
such as autovacuum_vacuum_scale_factor. These are the most important
individual tunables, which argues for putting them earlier than "Routine
Vacuuming".
Arguments for "after":
Although the discussion in "Routine Vacuuming" is rather involved, it is
arguably still introductory material that informs how the user will tune
autovacuum_vacuum_scale_factor. (Assuming that the user doesn't just go
by trial and error, which seems more likely in practice but not
necessarily the most useful working assumption for our purposes.)
---
doc/src/sgml/maintenance.sgml | 332 +++++++++++++++++-----------------
1 file changed, 166 insertions(+), 166 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 9cf9d030a..a6295c399 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -59,6 +59,172 @@
pleasant and productive experience with the system.
</para>
+ <sect1 id="autovacuum">
+ <title>The Autovacuum Daemon</title>
+
+ <indexterm>
+ <primary>autovacuum</primary>
+ <secondary>general information</secondary>
+ </indexterm>
+ <para>
+ <productname>PostgreSQL</productname> has an optional but highly
+ recommended feature called <firstterm>autovacuum</firstterm>,
+ whose purpose is to automate the execution of
+ <command>VACUUM</command> and <command>ANALYZE</command> commands.
+ When enabled, autovacuum checks for
+ tables that have had a large number of inserted, updated or deleted
+ tuples. These checks use the statistics collection facility;
+ therefore, autovacuum cannot be used unless <xref
+ linkend="guc-track-counts"/> is set to <literal>true</literal>.
+ In the default configuration, autovacuuming is enabled and the related
+ configuration parameters are appropriately set.
+ </para>
+
+ <para>
+ The <quote>autovacuum daemon</quote> actually consists of multiple processes.
+ There is a persistent daemon process, called the
+ <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
+ <firstterm>autovacuum worker</firstterm> processes for all databases. The
+ launcher will distribute the work across time, attempting to start one
+ worker within each database every <xref linkend="guc-autovacuum-naptime"/>
+ seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
+ a new worker will be launched every
+ <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
+ A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
+ are allowed to run at the same time. If there are more than
+ <varname>autovacuum_max_workers</varname> databases to be processed,
+ the next database will be processed as soon as the first worker finishes.
+ Each worker process will check each table within its database and
+ execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
+ <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
+ autovacuum workers' activity.
+ </para>
+
+ <para>
+ If several large tables all become eligible for vacuuming in a short
+ amount of time, all autovacuum workers might become occupied with
+ vacuuming those tables for a long period. This would result
+ in other tables and databases not being vacuumed until a worker becomes
+ available. There is no limit on how many workers might be in a
+ single database, but workers do try to avoid repeating work that has
+ already been done by other workers. Note that the number of running
+ workers does not count towards <xref linkend="guc-max-connections"/> or
+ <xref linkend="guc-superuser-reserved-connections"/> limits.
+ </para>
+
+ <para>
+ Tables whose <structfield>relfrozenxid</structfield> value is more than
+ <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
+ vacuumed (this also applies to those tables whose freeze max age has
+ been modified via storage parameters; see below). Otherwise, if the
+ number of tuples obsoleted since the last
+ <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
+ table is vacuumed. The vacuum threshold is defined as:
+<programlisting>
+vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
+</programlisting>
+ where the vacuum base threshold is
+ <xref linkend="guc-autovacuum-vacuum-threshold"/>,
+ the vacuum scale factor is
+ <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
+ and the number of tuples is
+ <structname>pg_class</structname>.<structfield>reltuples</structfield>.
+ </para>
+
+ <para>
+ The table is also vacuumed if the number of tuples inserted since the last
+ vacuum has exceeded the defined insert threshold, which is defined as:
+<programlisting>
+vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
+</programlisting>
+ where the vacuum insert base threshold is
+ <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
+ and vacuum insert scale factor is
+ <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
+ Such vacuums may allow portions of the table to be marked as
+ <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
+ can reduce the work required in subsequent vacuums.
+ For tables which receive <command>INSERT</command> operations but no or
+ almost no <command>UPDATE</command>/<command>DELETE</command> operations,
+ it may be beneficial to lower the table's
+ <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
+ tuples to be frozen by earlier vacuums. The number of obsolete tuples and
+ the number of inserted tuples are obtained from the cumulative statistics system;
+ it is a semi-accurate count updated by each <command>UPDATE</command>,
+ <command>DELETE</command> and <command>INSERT</command> operation. (It is
+ only semi-accurate because some information might be lost under heavy
+ load.) If the <structfield>relfrozenxid</structfield> value of the table
+ is more than <varname>vacuum_freeze_table_age</varname> transactions old,
+ an aggressive vacuum is performed to freeze old tuples and advance
+ <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
+ since the last vacuum are scanned.
+ </para>
+
+ <para>
+ For analyze, a similar condition is used: the threshold, defined as:
+<programlisting>
+analyze threshold = analyze base threshold + analyze scale factor * number of tuples
+</programlisting>
+ is compared to the total number of tuples inserted, updated, or deleted
+ since the last <command>ANALYZE</command>.
+ </para>
+
+ <para>
+ Partitioned tables are not processed by autovacuum. Statistics
+ should be collected by running a manual <command>ANALYZE</command> when it is
+ first populated, and again whenever the distribution of data in its
+ partitions changes significantly.
+ </para>
+
+ <para>
+ Temporary tables cannot be accessed by autovacuum. Therefore,
+ appropriate vacuum and analyze operations should be performed via
+ session SQL commands.
+ </para>
+
+ <para>
+ The default thresholds and scale factors are taken from
+ <filename>postgresql.conf</filename>, but it is possible to override them
+ (and many other autovacuum control parameters) on a per-table basis; see
+ <xref linkend="sql-createtable-storage-parameters"/> for more information.
+ If a setting has been changed via a table's storage parameters, that value
+ is used when processing that table; otherwise the global settings are
+ used. See <xref linkend="runtime-config-autovacuum"/> for more details on
+ the global settings.
+ </para>
+
+ <para>
+ When multiple workers are running, the autovacuum cost delay parameters
+ (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
+ <quote>balanced</quote> among all the running workers, so that the
+ total I/O impact on the system is the same regardless of the number
+ of workers actually running. However, any workers processing tables whose
+ per-table <literal>autovacuum_vacuum_cost_delay</literal> or
+ <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
+ are not considered in the balancing algorithm.
+ </para>
+
+ <para>
+ Autovacuum workers generally don't block other commands. If a process
+ attempts to acquire a lock that conflicts with the
+ <literal>SHARE UPDATE EXCLUSIVE</literal> lock held by autovacuum, lock
+ acquisition will interrupt the autovacuum. For conflicting lock modes,
+ see <xref linkend="table-lock-compatibility"/>. However, if the autovacuum
+ is running to prevent transaction ID wraparound (i.e., the autovacuum query
+ name in the <structname>pg_stat_activity</structname> view ends with
+ <literal>(to prevent wraparound)</literal>), the autovacuum is not
+ automatically interrupted.
+ </para>
+
+ <warning>
+ <para>
+ Regularly running commands that acquire locks conflicting with a
+ <literal>SHARE UPDATE EXCLUSIVE</literal> lock (e.g., ANALYZE) can
+ effectively prevent autovacuums from ever completing.
+ </para>
+ </warning>
+ </sect1>
+
<sect1 id="routine-vacuuming">
<title>Routine Vacuuming</title>
@@ -749,172 +915,6 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect3>
</sect2>
-
- <sect2 id="autovacuum">
- <title>The Autovacuum Daemon</title>
-
- <indexterm>
- <primary>autovacuum</primary>
- <secondary>general information</secondary>
- </indexterm>
- <para>
- <productname>PostgreSQL</productname> has an optional but highly
- recommended feature called <firstterm>autovacuum</firstterm>,
- whose purpose is to automate the execution of
- <command>VACUUM</command> and <command>ANALYZE</command> commands.
- When enabled, autovacuum checks for
- tables that have had a large number of inserted, updated or deleted
- tuples. These checks use the statistics collection facility;
- therefore, autovacuum cannot be used unless <xref
- linkend="guc-track-counts"/> is set to <literal>true</literal>.
- In the default configuration, autovacuuming is enabled and the related
- configuration parameters are appropriately set.
- </para>
-
- <para>
- The <quote>autovacuum daemon</quote> actually consists of multiple processes.
- There is a persistent daemon process, called the
- <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
- <firstterm>autovacuum worker</firstterm> processes for all databases. The
- launcher will distribute the work across time, attempting to start one
- worker within each database every <xref linkend="guc-autovacuum-naptime"/>
- seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
- a new worker will be launched every
- <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
- A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
- are allowed to run at the same time. If there are more than
- <varname>autovacuum_max_workers</varname> databases to be processed,
- the next database will be processed as soon as the first worker finishes.
- Each worker process will check each table within its database and
- execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
- <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
- autovacuum workers' activity.
- </para>
-
- <para>
- If several large tables all become eligible for vacuuming in a short
- amount of time, all autovacuum workers might become occupied with
- vacuuming those tables for a long period. This would result
- in other tables and databases not being vacuumed until a worker becomes
- available. There is no limit on how many workers might be in a
- single database, but workers do try to avoid repeating work that has
- already been done by other workers. Note that the number of running
- workers does not count towards <xref linkend="guc-max-connections"/> or
- <xref linkend="guc-superuser-reserved-connections"/> limits.
- </para>
-
- <para>
- Tables whose <structfield>relfrozenxid</structfield> value is more than
- <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
- vacuumed (this also applies to those tables whose freeze max age has
- been modified via storage parameters; see below). Otherwise, if the
- number of tuples obsoleted since the last
- <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
- table is vacuumed. The vacuum threshold is defined as:
-<programlisting>
-vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
-</programlisting>
- where the vacuum base threshold is
- <xref linkend="guc-autovacuum-vacuum-threshold"/>,
- the vacuum scale factor is
- <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
- and the number of tuples is
- <structname>pg_class</structname>.<structfield>reltuples</structfield>.
- </para>
-
- <para>
- The table is also vacuumed if the number of tuples inserted since the last
- vacuum has exceeded the defined insert threshold, which is defined as:
-<programlisting>
-vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
-</programlisting>
- where the vacuum insert base threshold is
- <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
- and vacuum insert scale factor is
- <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
- Such vacuums may allow portions of the table to be marked as
- <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
- can reduce the work required in subsequent vacuums.
- For tables which receive <command>INSERT</command> operations but no or
- almost no <command>UPDATE</command>/<command>DELETE</command> operations,
- it may be beneficial to lower the table's
- <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
- tuples to be frozen by earlier vacuums. The number of obsolete tuples and
- the number of inserted tuples are obtained from the cumulative statistics system;
- it is a semi-accurate count updated by each <command>UPDATE</command>,
- <command>DELETE</command> and <command>INSERT</command> operation. (It is
- only semi-accurate because some information might be lost under heavy
- load.) If the <structfield>relfrozenxid</structfield> value of the table
- is more than <varname>vacuum_freeze_table_age</varname> transactions old,
- an aggressive vacuum is performed to freeze old tuples and advance
- <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
- since the last vacuum are scanned.
- </para>
-
- <para>
- For analyze, a similar condition is used: the threshold, defined as:
-<programlisting>
-analyze threshold = analyze base threshold + analyze scale factor * number of tuples
-</programlisting>
- is compared to the total number of tuples inserted, updated, or deleted
- since the last <command>ANALYZE</command>.
- </para>
-
- <para>
- Partitioned tables are not processed by autovacuum. Statistics
- should be collected by running a manual <command>ANALYZE</command> when it is
- first populated, and again whenever the distribution of data in its
- partitions changes significantly.
- </para>
-
- <para>
- Temporary tables cannot be accessed by autovacuum. Therefore,
- appropriate vacuum and analyze operations should be performed via
- session SQL commands.
- </para>
-
- <para>
- The default thresholds and scale factors are taken from
- <filename>postgresql.conf</filename>, but it is possible to override them
- (and many other autovacuum control parameters) on a per-table basis; see
- <xref linkend="sql-createtable-storage-parameters"/> for more information.
- If a setting has been changed via a table's storage parameters, that value
- is used when processing that table; otherwise the global settings are
- used. See <xref linkend="runtime-config-autovacuum"/> for more details on
- the global settings.
- </para>
-
- <para>
- When multiple workers are running, the autovacuum cost delay parameters
- (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
- <quote>balanced</quote> among all the running workers, so that the
- total I/O impact on the system is the same regardless of the number
- of workers actually running. However, any workers processing tables whose
- per-table <literal>autovacuum_vacuum_cost_delay</literal> or
- <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
- are not considered in the balancing algorithm.
- </para>
-
- <para>
- Autovacuum workers generally don't block other commands. If a process
- attempts to acquire a lock that conflicts with the
- <literal>SHARE UPDATE EXCLUSIVE</literal> lock held by autovacuum, lock
- acquisition will interrupt the autovacuum. For conflicting lock modes,
- see <xref linkend="table-lock-compatibility"/>. However, if the autovacuum
- is running to prevent transaction ID wraparound (i.e., the autovacuum query
- name in the <structname>pg_stat_activity</structname> view ends with
- <literal>(to prevent wraparound)</literal>), the autovacuum is not
- automatically interrupted.
- </para>
-
- <warning>
- <para>
- Regularly running commands that acquire locks conflicting with a
- <literal>SHARE UPDATE EXCLUSIVE</literal> lock (e.g., ANALYZE) can
- effectively prevent autovacuums from ever completing.
- </para>
- </warning>
- </sect2>
</sect1>
--
2.40.1
[application/octet-stream] v3-0009-Overhaul-freezing-and-wraparound-docs.patch (65.0K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/4-v3-0009-Overhaul-freezing-and-wraparound-docs.patch)
download | inline diff:
From 27edfce6bc6414e79d79741248cfe5eae258e92a Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 13:04:13 -0700
Subject: [PATCH v3 9/9] Overhaul freezing and wraparound docs.
This is almost a complete rewrite. "Preventing Transaction ID
Wraparound Failures" becomes "Freezing to manage the transaction ID
space". This is follow-up work to commit 1de58df4, which added
page-level freezing to VACUUM.
The emphasis is now on the physical work of freezing pages. This flows
a little better than it otherwise would due to recent structural
cleanups to maintenance.sgml; discussion about freezing now immediately
follows discussion of cleanup of dead tuples. We still talk about the
problem of the system activating xidStopLimit protections in the same
section, but we use much less alarmist language about data corruption,
and are no longer overly concerned about the very worst case. We don't
rescind the recommendation that users recover from an xidStopLimit
outage by using single user mode, though that seems like something we
should aim to do in the near future.
There is no longer a separate sect3 to discuss MultiXactId related
issues. VACUUM now performs exactly the same processing steps when it
freezes a page, independent of the trigger condition.
Also describe the page-level freezing FPI optimization added by commit
1de58df4. This is expected to trigger the majority of all freezing with
many types of workloads.
---
doc/src/sgml/config.sgml | 20 +-
doc/src/sgml/logicaldecoding.sgml | 2 +-
doc/src/sgml/maintenance.sgml | 967 ++++++++++++++++------
doc/src/sgml/ref/create_table.sgml | 2 +-
doc/src/sgml/ref/prepare_transaction.sgml | 2 +-
doc/src/sgml/ref/vacuum.sgml | 6 +-
doc/src/sgml/ref/vacuumdb.sgml | 4 +-
doc/src/sgml/xact.sgml | 2 +-
8 files changed, 724 insertions(+), 281 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b56f073a9..a4ac4e740 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8359,7 +8359,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
Note that even when this parameter is disabled, the system
will launch autovacuum processes if necessary to
prevent transaction ID wraparound. See <xref
- linkend="vacuum-for-wraparound"/> for more information.
+ linkend="freezing-xid-space"/> for more information.
</para>
</listitem>
</varlistentry>
@@ -8548,7 +8548,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
This parameter can only be set at server start, but the setting
can be reduced for individual tables by
changing table storage parameters.
- For more information see <xref linkend="vacuum-for-wraparound"/>.
+ For more information see <xref linkend="freezing-xid-space"/>.
</para>
</listitem>
</varlistentry>
@@ -8577,7 +8577,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
400 million multixacts.
This parameter can only be set at server start, but the setting can
be reduced for individual tables by changing table storage parameters.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ For more information see <xref linkend="anti-wraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -9284,7 +9284,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
periodic manual <command>VACUUM</command> has a chance to run before an
anti-wraparound autovacuum is launched for the table. For more
information see
- <xref linkend="vacuum-for-wraparound"/>.
+ <xref linkend="aggressive-vacuum"/>.
</para>
</listitem>
</varlistentry>
@@ -9306,7 +9306,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
the value of <xref linkend="guc-autovacuum-freeze-max-age"/>, so
that there is not an unreasonably short time between forced
autovacuums. For more information see <xref
- linkend="vacuum-for-wraparound"/>.
+ linkend="freezing-xid-space"/>.
</para>
</listitem>
</varlistentry>
@@ -9343,7 +9343,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
set this value anywhere from zero to 2.1 billion,
<command>VACUUM</command> will silently adjust the effective
value to no less than 105% of <xref
- linkend="guc-autovacuum-freeze-max-age"/>.
+ linkend="guc-autovacuum-freeze-max-age"/>. For more
+ information see <xref linkend="xid-stop-limit"/>.
</para>
</listitem>
</varlistentry>
@@ -9367,7 +9368,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
<xref linkend="guc-autovacuum-multixact-freeze-max-age"/>, so that a
periodic manual <command>VACUUM</command> has a chance to run before an
anti-wraparound is launched for the table.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ For more information see <xref linkend="anti-wraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -9388,7 +9389,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
the value of <xref linkend="guc-autovacuum-multixact-freeze-max-age"/>,
so that there is not an unreasonably short time between forced
autovacuums.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ For more information see <xref linkend="freezing-xid-space"/>.
</para>
</listitem>
</varlistentry>
@@ -9421,7 +9422,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
this value anywhere from zero to 2.1 billion,
<command>VACUUM</command> will silently adjust the effective
value to no less than 105% of <xref
- linkend="guc-autovacuum-multixact-freeze-max-age"/>.
+ linkend="guc-autovacuum-multixact-freeze-max-age"/>. For more
+ information see <xref linkend="xid-stop-limit"/>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index cbd3aa804..80dade3be 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -353,7 +353,7 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
because neither required WAL nor required rows from the system catalogs
can be removed by <command>VACUUM</command> as long as they are required by a replication
slot. In extreme cases this could cause the database to shut down to prevent
- transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ transaction ID wraparound (see <xref linkend="freezing-xid-space"/>).
So if a slot is no longer required it should be dropped.
</para>
</caution>
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 5546d8c7d..a480e4f8e 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -148,13 +148,8 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac
<xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
Such vacuums may allow portions of the table to be marked as
<firstterm>all visible</firstterm> and also allow tuples to be frozen, which
- can reduce the work required in subsequent vacuums.
- For tables which receive <command>INSERT</command> operations but no or
- almost no <command>UPDATE</command>/<command>DELETE</command> operations,
- it may be beneficial to lower the table's
- <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
- tuples to be frozen by earlier vacuums. The number of obsolete tuples and
- the number of inserted tuples are obtained from the cumulative statistics system;
+ can reduce the work required in subsequent vacuums. The number of obsolete tuples
+ and the number of inserted tuples are obtained from the cumulative statistics system;
it is a semi-accurate count updated by each <command>UPDATE</command>,
<command>DELETE</command> and <command>INSERT</command> operation. (It is
only semi-accurate because some information might be lost under heavy
@@ -273,15 +268,20 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ <simpara>To maintain the system's ability to allocated new
+ transaction IDs through freezing.</simpara>
</listitem>
<listitem>
<simpara>To update the visibility map, which speeds
up <link linkend="indexes-index-only-scans">index-only
- scans</link>.</simpara>
+ scans</link>, and helps the next <command>VACUUM</command>
+ operation avoid needlessly scanning already-frozen pages.</simpara>
+ </listitem>
+
+ <listitem>
+ <simpara>To truncate obsolescent transaction status information,
+ when possible.</simpara>
</listitem>
<listitem>
@@ -483,302 +483,671 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</warning>
</sect2>
- <sect2 id="vacuum-for-wraparound">
- <title>Preventing Transaction ID Wraparound Failures</title>
-
- <indexterm zone="vacuum-for-wraparound">
- <primary>transaction ID</primary>
- <secondary>wraparound</secondary>
- </indexterm>
+ <sect2 id="freezing-xid-space">
+ <title>Freezing to manage the transaction ID space</title>
<indexterm>
- <primary>wraparound</primary>
- <secondary>of transaction IDs</secondary>
+ <primary>Freezing</primary>
+ <secondary>of transaction IDs and MultiXact IDs</secondary>
</indexterm>
<para>
- <productname>PostgreSQL</productname>'s <link
- linkend="mvcc-intro">MVCC</link> transaction semantics depend on
- being able to compare <glossterm linkend="glossary-xid">transaction
- ID numbers (<acronym>XID</acronym>)</glossterm> to determine
- whether or not the row is visible to each query's MVCC snapshot
- (see <xref linkend="interpreting-xid-stamps"/>). But since
- on-disk storage of transaction IDs in heap pages uses a truncated
- 32-bit representation to save space (rather than the full 64-bit
- representation), it is necessary to vacuum every table in every
- database <emphasis>at least</emphasis> once every two billion
- transactions (though far more frequent vacuuming is typical).
+ <command>VACUUM</command> often marks some of the pages that it
+ scans <emphasis>frozen</emphasis>, indicating that all eligible
+ rows on the page were inserted by a transaction that committed
+ sufficiently far in the past that the effects of the inserting
+ transaction are certain to be visible to all current and future
+ transactions. The specific Transaction ID number
+ (<acronym>XID</acronym>) stored in a frozen heap row's
+ <structfield>xmin</structfield> field is no longer needed to
+ determine anything about the row's visibility. Furthermore, when
+ a row undergoing freezing happens to have an XID set in its
+ <structfield>xmax</structfield> field (possibly an XID left behind
+ by an earlier <command>SELECT FOR UPDATE</command> row locker),
+ the <structfield>xmax</structfield> field's XID is usually also
+ removed.
</para>
<para>
- <xref linkend="guc-vacuum-freeze-min-age"/>
- controls how old an XID value has to be before rows bearing that XID will be
- frozen. Increasing this setting may avoid unnecessary work if the
- rows that would otherwise be frozen will soon be modified again,
- but decreasing this setting increases
- the number of transactions that can elapse before the table must be
- vacuumed again.
+ Once frozen, heap pages are <quote>self-contained</quote>. Every
+ query can read all of the page's rows in a way that assumes that
+ the inserting transaction committed and is visible to its
+ <acronym>MVCC</acronym> snapshot. No query will ever have to
+ consult external transaction status metadata to interpret the
+ page's contents, either. In particular,
+ <filename>pg_xact</filename> transaction XID commit/abort status
+ lookups won't take place during query execution.
</para>
<para>
- <command>VACUUM</command> uses the <link linkend="storage-vm">visibility map</link>
- to determine which pages of a table must be scanned. Normally, it
- will skip pages that don't have any dead row versions even if those pages
- might still have row versions with old XID values. Therefore, normal
- <command>VACUUM</command>s won't always freeze every old row version in the table.
- When that happens, <command>VACUUM</command> will eventually need to perform an
- <firstterm>aggressive vacuum</firstterm>, which will freeze all eligible unfrozen
- XID and MXID values, including those from all-visible but not all-frozen pages.
- In practice most tables require periodic aggressive vacuuming.
- <xref linkend="guc-vacuum-freeze-table-age"/>
- controls when <command>VACUUM</command> does that: all-visible but not all-frozen
- pages are scanned if the number of transactions that have passed since the
- last such scan is greater than <varname>vacuum_freeze_table_age</varname> minus
- <varname>vacuum_freeze_min_age</varname>. Setting
- <varname>vacuum_freeze_table_age</varname> to 0 forces <command>VACUUM</command> to
- always use its aggressive strategy.
+ Freezing is a <acronym>WAL</acronym>-logged operation, so when
+ <command>VACUUM</command> freezes a heap page, any copy of the
+ page located on a physical replication standby server will itself
+ be <quote>frozen</quote> shortly thereafter (when the relevant
+ <literal>FREEZE_PAGE</literal> <acronym>WAL</acronym> record is
+ replayed on the standby). Queries that run on physical
+ replication standbys thereby avoid <filename>pg_xact</filename>
+ lookups when reading from frozen pages, in just the same way as
+ queries that run on the primary server
+ <footnote>
+ <para>
+ In this regard freezing is unlike setting transaction status
+ <quote>hint bits</quote> in tuple headers: setting hint bits
+ doesn't usually need to be <acronym>WAL</acronym>-logged, and
+ can take place on physical replication standby servers without
+ the involvement of the primary server. The purpose of hint bits
+ is to avoid repeat <filename>pg_xact</filename> lookups for the
+ same tuples, strictly as an optimization. The purpose of
+ freezing (from the point of view of individual tuples) is to
+ <emphasis>reliably</emphasis> remove each tuple's dependency on
+ <filename>pg_xact</filename>, ultimately making it safe to
+ truncate <filename>pg_xact</filename> from time to time.
+ </para>
+ </footnote>.
</para>
<para>
- The maximum time that a table can go unvacuumed is two billion
- transactions minus the <varname>vacuum_freeze_min_age</varname> value at
- the time of the last aggressive vacuum. If it were to go
- unvacuumed for longer than
- that, data loss could result. To ensure that this does not happen,
- autovacuum is invoked on any table that might contain unfrozen rows with
- XIDs older than the age specified by the configuration parameter <xref
- linkend="guc-autovacuum-freeze-max-age"/>. (This will happen even if
- autovacuum is disabled.)
+ It can be useful for <command>VACUUM</command> to put off some of
+ the work of freezing, but <command>VACUUM</command> cannot put off
+ freezing forever. Since on-disk storage of transaction IDs in
+ heap row headers uses a truncated 32-bit representation to save
+ space (rather than the full 64-bit representation), freezing plays
+ a crucial role in enabling <link
+ linkend="aggressive-vacuum">management of the XID address
+ space</link> by <command>VACUUM</command>. If, for whatever
+ reason, <command>VACUUM</command> is unable to freeze older XIDs
+ on behalf of an application that continues to require new XID
+ allocations, the system will eventually
+ <link linkend="xid-stop-limit">refuse to allocate new transaction IDs</link>.
+ The system generally only enters this state when autovacuum is
+ misconfigured.
</para>
<para>
- This implies that if a table is not otherwise vacuumed,
- autovacuum will be invoked on it approximately once every
- <varname>autovacuum_freeze_max_age</varname> minus
- <varname>vacuum_freeze_min_age</varname> transactions.
- For tables that are regularly vacuumed for space reclamation purposes,
- this is of little importance. However, for static tables
- (including tables that receive inserts, but no updates or deletes),
- there is no need to vacuum for space reclamation, so it can
- be useful to try to maximize the interval between forced autovacuums
- on very large static tables. Obviously one can do this either by
- increasing <varname>autovacuum_freeze_max_age</varname> or decreasing
- <varname>vacuum_freeze_min_age</varname>.
+ <xref linkend="guc-vacuum-freeze-min-age"/> controls when freezing
+ takes place. When <command>VACUUM</command> scans a heap page
+ containing even one XID that has already attained an age exceeding
+ this value, the page is frozen.
+ </para>
+
+ <indexterm>
+ <primary>MultiXact ID</primary>
+ <secondary>Freezing of</secondary>
+ </indexterm>
+
+ <para>
+ <firstterm>MultiXact IDs</firstterm> are used to support row
+ locking by multiple transactions. Since there is only limited
+ space in a tuple header to store lock information, that
+ information is encoded as a <quote>multiple transaction
+ ID</quote>, or MultiXact ID for short, whenever there is more
+ than one transaction concurrently locking a row. Information
+ about which transaction IDs are included in any particular
+ MultiXact ID is stored separately in
+ <filename>pg_multixact</filename>, and only the MultiXact ID
+ itself (a 32-bit unsigned integer) appears in the tuple's
+ <structfield>xmax</structfield> field. This creates a dependency
+ on external transaction status information similar to the
+ dependency that ordinary unfrozen XIDs have on commit status
+ information stored in <filename>pg_xact</filename>.
+ <command>VACUUM</command> must therefore occasionally remove
+ MultiXact IDs from tuples during freezing.
</para>
<para>
- The effective maximum for <varname>vacuum_freeze_table_age</varname> is 0.95 *
- <varname>autovacuum_freeze_max_age</varname>; a setting higher than that will be
- capped to the maximum. A value higher than
- <varname>autovacuum_freeze_max_age</varname> wouldn't make sense because an
- anti-wraparound autovacuum would be triggered at that point anyway, and
- the 0.95 multiplier leaves some breathing room to run a manual
- <command>VACUUM</command> before that happens. As a rule of thumb,
- <command>vacuum_freeze_table_age</command> should be set to a value somewhat
- below <varname>autovacuum_freeze_max_age</varname>, leaving enough gap so that
- a regularly scheduled <command>VACUUM</command> or an autovacuum triggered by
- normal delete and update activity is run in that window. Setting it too
- close could lead to anti-wraparound autovacuums, even though the table
- was recently vacuumed to reclaim space, whereas lower values lead to more
- frequent aggressive vacuuming.
+ <xref linkend="guc-vacuum-multixact-freeze-min-age"/> also
+ controls when freezing takes place. It is analogous to
+ <varname>vacuum_freeze_min_age</varname>, but <quote>age</quote>
+ is expressed in units of MultiXact ID.
+ Lowering <varname>vacuum_multixact_freeze_min_age</varname>
+ <emphasis>forces</emphasis> <command>VACUUM</command> to process
+ <structfield>xmax</structfield> fields containing a MultiXact ID
+ in cases where it would otherwise opt to put off the work of
+ processing <structfield>xmax</structfield> until the next
+ <command>VACUUM</command> <footnote>
+ <para>
+ <quote>Freezing</quote> of <structfield>xmax</structfield>
+ fields (whether they were found to contain an XID or a MultiXact
+ ID) generally means clearing <structfield>xmax</structfield>.
+ <command>VACUUM</command> may occasionally encounter an
+ individual MultiXact ID that must be removed to advance
+ <structfield>relminmxid</structfield> by the required amount,
+ which can only be processed by generating a replacement
+ MultiXact ID (containing just the non-removable subset of member
+ XIDs from the original MultiXact ID), and then setting the
+ tuple's <structfield>xmax</structfield> to the new/replacement
+ MultiXact ID value.
+ </para>
+ </footnote>. The setting generally doesn't significantly
+ influence the total number of pages <command>VACUUM</command>
+ freezes, even in tables that contain relatively many MultiXact
+ IDs. This is because <command>VACUUM</command> generally prefers
+ proactively processing for most individual
+ <structfield>xmax</structfield> fields that contain a MultiXact ID
+ (eager proactive processing is typically cheaper).
</para>
<para>
- The sole disadvantage of increasing <varname>autovacuum_freeze_max_age</varname>
- (and <varname>vacuum_freeze_table_age</varname> along with it) is that
- the <filename>pg_xact</filename> and <filename>pg_commit_ts</filename>
- subdirectories of the database cluster will take more space, because it
- must store the commit status and (if <varname>track_commit_timestamp</varname> is
- enabled) timestamp of all transactions back to
- the <varname>autovacuum_freeze_max_age</varname> horizon. The commit status uses
- two bits per transaction, so if
- <varname>autovacuum_freeze_max_age</varname> is set to its maximum allowed value
- of two billion, <filename>pg_xact</filename> can be expected to grow to about half
- a gigabyte and <filename>pg_commit_ts</filename> to about 20GB. If this
- is trivial compared to your total database size,
- setting <varname>autovacuum_freeze_max_age</varname> to its maximum allowed value
- is recommended. Otherwise, set it depending on what you are willing to
- allow for <filename>pg_xact</filename> and <filename>pg_commit_ts</filename> storage.
- (The default, 200 million transactions, translates to about 50MB
- of <filename>pg_xact</filename> storage and about 2GB of <filename>pg_commit_ts</filename>
- storage.)
+ Managing the added <acronym>WAL</acronym> volume from freezing
+ over time is an important consideration for
+ <command>VACUUM</command>. It is why <command>VACUUM</command>
+ doesn't just freeze every eligible tuple at the earliest
+ opportunity: the <acronym>WAL</acronym> written to freeze a page's
+ tuples <quote>goes to waste</quote> in cases where the resulting
+ frozen tuples are soon deleted or updated anyway. It's also why
+ <command>VACUUM</command> <emphasis>will</emphasis> freeze all
+ eligible tuples from a heap page once the decision to freeze at
+ least one tuple is taken: at that point the added cost to freeze
+ all eligible tuples eagerly (measured in <quote>extra bytes of
+ <acronym>WAL</acronym> written</quote>) is far lower than the
+ probable cost of deferring freezing until a future
+ <command>VACUUM</command> operation against the same table.
+ Furthermore, once the page is frozen it can generally be marked as
+ all-frozen in the visibility map right away.
</para>
- <para>
- One disadvantage of decreasing <varname>vacuum_freeze_min_age</varname> is that
- it might cause <command>VACUUM</command> to do useless work: freezing a row
- version is a waste of time if the row is modified
- soon thereafter (causing it to acquire a new XID). So the setting should
- be large enough that rows are not frozen until they are unlikely to change
- any more.
- </para>
+ <note>
+ <para>
+ In <productname>PostgreSQL</productname> versions before 16,
+ <command>VACUUM</command> triggered freezing at the level of
+ individual <structfield>xmin</structfield> and
+ <structfield>xmax</structfield> fields. Freezing only affected
+ the exact XIDs that had already attained an age of
+ <varname>vacuum_freeze_min_age</varname> or greater.
+ </para>
+ </note>
<para>
- To track the age of the oldest unfrozen XIDs in a database,
- <command>VACUUM</command> stores XID
- statistics in the system tables <structname>pg_class</structname> and
- <structname>pg_database</structname>. In particular,
- the <structfield>relfrozenxid</structfield> column of a table's
- <structname>pg_class</structname> row contains the oldest remaining unfrozen
- XID at the end of the most recent <command>VACUUM</command> that successfully
- advanced <structfield>relfrozenxid</structfield> (typically the most recent
- aggressive VACUUM). Similarly, the
- <structfield>datfrozenxid</structfield> column of a database's
- <structname>pg_database</structname> row is a lower bound on the unfrozen XIDs
- appearing in that database — it is just the minimum of the
- per-table <structfield>relfrozenxid</structfield> values within the database.
- A convenient way to
- examine this information is to execute queries such as:
-
-<programlisting>
-SELECT c.oid::regclass as table_name,
- greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age
-FROM pg_class c
-LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
-WHERE c.relkind IN ('r', 'm');
-
-SELECT datname, age(datfrozenxid) FROM pg_database;
-</programlisting>
-
- The <literal>age</literal> column measures the number of transactions from the
- cutoff XID to the current transaction's XID.
+ <command>VACUUM</command> also triggers freezing of a page in
+ cases where it already proved necessary to write out a full page
+ image (<acronym>FPI</acronym>) as part of a <acronym>WAL</acronym>
+ record describing how dead tuples were removed <footnote>
+ <para>
+ Actually, the <quote>freeze on an <acronym>FPI</acronym>
+ write</quote> mechanism isn't just triggered whenever
+ <command>VACUUM</command> needed to write an
+ <acronym>FPI</acronym> for torn page protection as part of
+ writing a <literal>PRUNE</literal> <acronym>WAL</acronym> record
+ describing how dead tuples were removed. The
+ <acronym>FPI</acronym> mechanism can also be triggered when hint
+ bits are set by <command>VACUUM</command>, if and only if doing
+ so necessitates writing an <acronym>FPI</acronym>.
+ <acronym>WAL</acronym>-logging in order to set hint bits is only
+ possible when the <xref linkend="guc-wal-log-hints"/> option is
+ enabled in <filename>postgresql.conf</filename>, or when data
+ checksums were enabled when the cluster was initialized with
+ <xref linkend="app-initdb"/>.
+ </para>
+ </footnote> (see <xref linkend="wal-reliability"/> for background
+ information about how <acronym>FPI</acronym>s provide torn page
+ protection). This <quote>freeze on an <acronym>FPI</acronym>
+ write</quote> batching mechanism often avoids the need for some
+ future <command>VACUUM</command> operation to write an additional
+ <acronym>FPI</acronym> for the same page as part of a
+ <acronym>WAL</acronym> record describing how live tuples were
+ frozen. In effect, <command>VACUUM</command> writes slightly more
+ <acronym>WAL</acronym> in the short term with the aim of
+ ultimately needing to write much less <acronym>WAL</acronym> in
+ the long term.
</para>
<tip>
<para>
- When the <command>VACUUM</command> command's <literal>VERBOSE</literal>
- parameter is specified, <command>VACUUM</command> prints various
- statistics about the table. This includes information about how
- <structfield>relfrozenxid</structfield> and
- <structfield>relminmxid</structfield> advanced, and the number of
- newly frozen pages. The same details appear in the server log when
- autovacuum logging (controlled by <xref
- linkend="guc-log-autovacuum-min-duration"/>) reports on a
- <command>VACUUM</command> operation executed by autovacuum.
+ For tables which receive <command>INSERT</command> operations,
+ but few or no <command>UPDATE</command>/<command>DELETE</command>
+ operations, it may be beneficial to selectively lower <xref
+ linkend="reloption-autovacuum-freeze-min-age"/> for the table.
+ <command>VACUUM</command> may thereby be able to freeze the
+ table's pages <quote>eagerly</quote> during earlier autovacuums
+ triggered by <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
</para>
</tip>
- <para>
- <command>VACUUM</command> normally only scans pages that have been modified
- since the last vacuum, but <structfield>relfrozenxid</structfield> can only be
- advanced when every page of the table
- that might contain unfrozen XIDs is scanned. This happens when
- <structfield>relfrozenxid</structfield> is more than
- <varname>vacuum_freeze_table_age</varname> transactions old, when
- <command>VACUUM</command>'s <literal>FREEZE</literal> option is used, or when all
- pages that are not already all-frozen happen to
- require vacuuming to remove dead row versions. When <command>VACUUM</command>
- scans every page in the table that is not already all-frozen, it should
- set <literal>age(relfrozenxid)</literal> to a value just a little more than the
- <varname>vacuum_freeze_min_age</varname> setting
- that was used (more by the number of transactions started since the
- <command>VACUUM</command> started). <command>VACUUM</command>
- will set <structfield>relfrozenxid</structfield> to the oldest XID
- that remains in the table, so it's possible that the final value
- will be much more recent than strictly required.
- If no <structfield>relfrozenxid</structfield>-advancing
- <command>VACUUM</command> is issued on the table until
- <varname>autovacuum_freeze_max_age</varname> is reached, an autovacuum will soon
- be forced for the table.
- </para>
+ <caution>
+ <para>
+ <command>VACUUM</command> may not be able to freeze every tuple's
+ <structfield>xmin</structfield> in relatively rare cases. The
+ criteria that determines basic eligibility for freezing is the
+ same as the one that determines if a deleted tuple can be
+ removed: the XID-based <literal>removable cutoff</literal> that
+ appears in the server log's autovacuum log reports (controlled by
+ <xref linkend="guc-log-autovacuum-min-duration"/>).
+ </para>
+ <para>
+ In extreme cases, a long-running transaction can hold back every
+ <command>VACUUM</command>'s removable cutoff for so long that the
+ system is forced to activate <link
+ linkend="xid-stop-limit"><literal>xidStopLimit</literal> mode
+ protections</link>.
+ </para>
+ </caution>
- <para>
- If for some reason autovacuum fails to clear old XIDs from a table, the
- system will begin to emit warning messages like this when the database's
- oldest XIDs reach forty million transactions from the wraparound point:
+ <sect3 id="aggressive-vacuum">
+ <title>Aggressive <command>VACUUM</command></title>
+
+ <indexterm zone="aggressive-vacuum">
+ <primary>transaction ID</primary>
+ <secondary>wraparound</secondary>
+ </indexterm>
+
+ <indexterm>
+ <primary>wraparound</primary>
+ <secondary>of transaction IDs and MultiXact IDs</secondary>
+ </indexterm>
+
+ <para>
+ As noted already, freezing doesn't just allow queries to avoid
+ lookups of subsidiary transaction status information in
+ structures such as <filename>pg_xact</filename>. Freezing also
+ plays a crucial role in enabling management of the XID address
+ space by <command>VACUUM</command>. <command>VACUUM</command>
+ maintains information about the oldest unfrozen XID that remains
+ in the table when it uses its <firstterm>aggressive strategy</firstterm>.
+ </para>
+
+ <para>
+ Aggressive <command>VACUUM</command> will update the table's
+ <structname>pg_class</structname>.<structfield>relfrozenxid</structfield>
+ to the value that it determined to be the oldest remaining XID;
+ the table's <structfield>relfrozenxid</structfield>
+ <quote>advances</quote> by a certain number of XIDs. Aggressive
+ <command>VACUUM</command> may also need to update the
+ <structfield>datfrozenxid</structfield> column of the database's
+ <structname>pg_database</structname> row in turn.
+ <structfield>datfrozenxid</structfield> is a lower bound on the
+ unfrozen XIDs appearing in that database — it is just the
+ minimum of the per-table <structfield>relfrozenxid</structfield>
+ values (the <structfield>relfrozenxid</structfield> that has
+ attained the greatest age) within the database.
+ </para>
+
+ <para>
+ Aggressive <command>VACUUM</command> also maintains the
+ <structname>pg_class</structname>.<structfield>relminmxid</structfield>
+ and <structname>pg_database</structname>.<structfield>datminmxid</structfield>
+ fields. These are needed to track the oldest MultiXact ID that
+ remains in the table and database, respectively.
+ </para>
+
+ <para>
+ The extra steps performed within every aggressive
+ <command>VACUUM</command> against every table have the overall
+ effect of tracking the oldest remaining unfrozen transaction ID
+ in the entire cluster (every table from every database).
+ Aggressive <command>VACUUM</command>s will (in the aggregate and
+ over time) make sure that the oldest unfrozen transaction ID in
+ the entire system is never too far in the past.
+ </para>
+
+ <note>
+ <title>Managing the Transaction ID Space</title>
+ <para>
+ Freezing removes <emphasis>local</emphasis> dependencies on
+ external transaction status information from individual heap
+ pages. Advancing <structfield>relfrozenxid</structfield>
+ removes <emphasis>global</emphasis> dependencies from whole
+ tables in turn.
+ </para>
+ <para>
+ The oldest XID in the entire cluster can be thought of as the
+ beginning of the XID space, while the next unallocated XID can
+ be thought of as the end of the XID space. This space
+ represents the range of XIDs that might still require
+ transaction commit/abort status lookups in <filename>pg_xact</filename>.
+ </para>
+ </note>
+
+ <para>
+ The maximum XID age that the system can tolerate (i.e., the
+ maximum <quote>distance</quote> between the oldest unfrozen
+ transaction ID in any table according to
+ <structname>pg_class</structname>.<structfield>relfrozenxid</structfield>,
+ and the next unallocated transaction ID) is about 2.1 billion
+ transaction IDs. This <quote>maximum XID age</quote> invariant
+ makes it fundamentally impossible to put off aggressive
+ <command>VACUUM</command>s (and freezing) forever
+ <footnote>
+ <para>
+ Aggressive <command>VACUUM</command>s cannot be put off
+ forever, <emphasis>barring the edge-case where the
+ installation is never expected to consume more than about 2.1
+ billion XIDs</emphasis>. In practice this has practical
+ relevance.
+ </para>
+ </footnote>. The invariant imposes an absolute hard limit on how
+ long any table can go without an aggressive <command>VACUUM</command>.
+ </para>
+
+ <para>
+ If the hard limit is ever reached, then the system will activate
+ <link linkend="xid-stop-limit"><literal>xidStopLimit</literal>
+ mode</link>, which temporarily prevents the allocation of new
+ permanent transaction IDs. The system will only deactive
+ <literal>xidStopLimit</literal> mode when
+ <command>VACUUM</command> (typically run by autovacuum) succeeds
+ in advancing the oldest <literal>datfrozenxid</literal> in the
+ cluster (via an aggressive <command>VACUUM</command> that runs to
+ completion against the table that has the oldest
+ <structfield>relfrozenxid</structfield>).
+ </para>
+
+ <para>
+ The 2.1 billion XIDs <quote>maximum XID age</quote> invariant
+ must be preserved because transaction IDs stored in heap row
+ headers use a truncated 32-bit representation (rather than the
+ full 64-bit representation). Since all unfrozen transaction IDs
+ from heap tuple headers <emphasis>must</emphasis> be from the
+ same transaction ID epoch (or from a space in the 64-bit
+ representation that spans two adjoining transaction ID epochs),
+ there isn't any need to store a separate epoch field in each
+ tuple header (see <xref linkend="interpreting-xid-stamps"/> for
+ further details). This scheme has the advantage of requiring
+ much less on-disk storage space than a design that stores an XID
+ epoch alongside each XID stored in each heap tuple header. It
+ has the disadvantage of constraining the system's ability to
+ allocate new XIDs in the worst case scenario where
+ <literal>xidStopLimit</literal> mode is used to preserve the
+ <quote>maximum XID age</quote> invariant.
+ </para>
+
+ <para>
+ There is only one major runtime behavioral difference between
+ aggressive mode <command>VACUUM</command>s and non-aggressive
+ <command>VACUUM</command>s: only non-aggressive
+ <command>VACUUM</command>s will skip pages that don't have any
+ dead row versions even if those pages still have row versions
+ with old XID values (pages marked as all-visible in the
+ visibility map). Aggressive <command>VACUUM</command>s can only
+ skip pages that are marked as both all-visible and all-frozen.
+ Consequently, non-aggressive <command>VACUUM</command>s usually
+ won't freeze <emphasis>every</emphasis> page containing an XID
+ that has already attained an age of
+ <varname>vacuum_freeze_min_age</varname> or more. Failing to
+ freeze older pages during non-aggressive
+ <command>VACUUM</command>s may lead to aggressive
+ <command>VACUUM</command>s that perform a disproportionately
+ large amount of the work of freezing required by one particular
+ table.
+ </para>
+
+ <tip>
+ <para>
+ When the <command>VACUUM</command> command's
+ <literal>VERBOSE</literal> parameter is specified,
+ <command>VACUUM</command> prints various statistics about the
+ table. Its output includes information about how
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> advanced, and the number
+ of newly frozen pages. The same details appear in the server
+ log when autovacuum logging (controlled by <xref
+ linkend="guc-log-autovacuum-min-duration"/>) reports on a
+ <command>VACUUM</command> operation executed by autovacuum.
+ </para>
+ </tip>
+
+ <note>
+ <para>
+ In practice, most tables require periodic aggressive vacuuming.
+ However, some individual non-aggressive
+ <command>VACUUM</command> operations may be able to advance
+ <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield>. Non-aggressive
+ <structfield>relfrozenxid</structfield>/<structfield>relminmxid</structfield>
+ advancement is most common in small, frequently modified tables.
+ </para>
+ </note>
+
+ <para>
+ Most individual tables will eventually need an aggressive
+ <command>VACUUM</command>, which will reliably freeze all pages
+ with XID (or MultiXact ID) values older than
+ <varname>vacuum_freeze_min_age</varname> (or older than
+ <varname>vacuum_multixact_freeze_min_age</varname>), including
+ those from all-visible but not all-frozen pages (and then advance
+ <structname>pg_class</structname>.<structfield>relfrozenxid</structfield>
+ to a value that reflects all that). <xref
+ linkend="guc-vacuum-freeze-table-age"/> controls when
+ <command>VACUUM</command> must use its aggressive strategy. If
+ <literal>age(relfrozenxid)</literal> exceeds
+ <varname>vacuum_freeze_table_age</varname> at the start of
+ <command>VACUUM</command>, that <command>VACUUM</command> will
+ use the aggressive strategy; otherwise the standard
+ non-aggressive strategy is used. Setting
+ <varname>vacuum_freeze_table_age</varname> to 0 forces
+ <command>VACUUM</command> to always use its aggressive strategy.
+ </para>
+ </sect3>
+
+ <sect3 id="anti-wraparound-autovacuums">
+ <title>Anti-Wraparound Autovacuums</title>
+
+ <para>
+ To ensure that every table has its
+ <structfield>relfrozenxid</structfield> advanced at somewhat
+ regular intervals, even in the case of completely static tables,
+ autovacuum runs against any table that might contain unfrozen
+ rows with XIDs older than the age specified by the configuration
+ parameter <xref linkend="guc-autovacuum-freeze-max-age"/>. These
+ are <firstterm>anti-wraparound autovacuums</firstterm>.
+ Anti-wraparound autovacuums can happen even when autovacuum is
+ nominally disabled in <filename>postgresql.conf</filename>.
+ </para>
+
+ <para>
+ In practice, all anti-wraparound autovacuums will use
+ <command>VACUUM</command>'s aggressive strategy (if they didn't,
+ then it would defeat the whole purpose of anti-wraparound
+ autovacuuming). Use of <command>VACUUM</command>'s aggressive
+ strategy is certain, because the effective value of
+ <varname>vacuum_freeze_table_age</varname> is silently
+ <quote>clamped</quote> to a value no greater than 95% of the
+ current value of <varname>autovacuum_freeze_max_age</varname>.
+ </para>
+
+ <para>
+ As a rule of thumb, <command>vacuum_freeze_table_age</command>
+ should be set to a value somewhat below
+ <varname>autovacuum_freeze_max_age</varname>, so that there is a
+ window during which any autovacuum triggered by inserts, updates,
+ or deletes (or any manually issued <command>VACUUM</command>)
+ will become an aggressive <command>VACUUM</command>. Such
+ <command>VACUUM</command>s will reliably advance
+ <structfield>relfrozenxid</structfield> in passing, even though
+ autovacuum won't have specifically set out to make sure
+ <structfield>relfrozenxid</structfield> advances through
+ anti-wraparound autovacuuming. Anti-wraparound autovacuums may
+ never be required at all in tables that regularly require
+ vacuuming to <link linkend="vacuum-for-space-recovery">reclaim
+ space from dead tuples</link> and/or to <link
+ linkend="vacuum-for-visibility-map">set pages all-visible in the
+ visibility map</link> (especially if
+ <varname>vacuum_freeze_table_age</varname> is set to a value
+ significantly below
+ <varname>autovacuum_freeze_max_age</varname>).
+ </para>
+
+ <note>
+ <title>Note on terminology</title>
+ <para>
+ Aggressive <command>VACUUM</command> is a special form of
+ <command>VACUUM</command>. An aggressive
+ <command>VACUUM</command> must advance
+ <structfield>relfrozenxid</structfield> up to an XID value that
+ is no greater than <varname>vacuum_freeze_min_age</varname> XIDs
+ in age as of the <emphasis>start</emphasis> of the
+ <command>VACUUM</command> operation.
+ </para>
+ <para>
+ Anti-wraparound autovacuum is a special form of Autovacuum. Its
+ purpose is to make sure that
+ <structfield>relfrozenxid</structfield> is advanced when no
+ earlier aggressive <command>VACUUM</command> ran and advanced
+ <structfield>relfrozenxid</structfield> in passing (often
+ because no <command>VACUUM</command> needed to run against the
+ table at all).
+ </para>
+ <para>
+ There is only one runtime behavioral difference between
+ anti-wraparound autovacuums and other autovacuums that happen to
+ end up running an aggressive <command>VACUUM</command>:
+ Anti-wraparound autovacuums <emphasis>cannot be
+ autocancelled</emphasis>. This means that autovacuum workers
+ that perform anti-wraparound autovacuuming do not yield to
+ conflicting relation-level lock requests (e.g., from
+ <command>ALTER TABLE</command>). See <xref
+ linkend="autovacuum-lock-conflicts"/> for a full explanation.
+ </para>
+ </note>
+
+ <para>
+ <command>VACUUM</command> also applies <xref
+ linkend="guc-vacuum-multixact-freeze-table-age"/> and <xref
+ linkend="guc-autovacuum-multixact-freeze-max-age"/>. These are
+ independent MultiXact ID based triggers of aggressive
+ <command>VACUUM</command> (and anti-wraparound autovacuum). They
+ are applied by following rules analogous to the rules already
+ described for <varname>vacuum_freeze_table_age</varname> and
+ <varname>autovacuum_freeze_max_age</varname>, respectively
+ <footnote>
+ <para>
+ Though note that autovacuum (and <command>VACUUM</command>) use a lower
+ <quote>effective</quote>
+ <varname>autovacuum_multixact_freeze_max_age</varname>
+ value (determined dynamically) to deal with issues with
+ truncation of the <acronym>SLRU</acronym> storage areas, as
+ explained in <xref linkend="vacuum-truncate-xact-status"/>
+ </para>
+ </footnote>.
+ </para>
+
+ <para>
+ It doesn't matter if it was <varname>vacuum_freeze_table_age</varname> or
+ <varname>vacuum_multixact_freeze_table_age</varname> that
+ triggered <command>VACUUM</command>'s decision to use its
+ aggressive strategy. <emphasis>Every</emphasis> aggressive
+ <command>VACUUM</command> will advance
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> by following the same
+ generic steps at runtime.
+ </para>
+
+ <para>
+ A convenient way to examine information about
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> is to execute queries such as:
+
+<programlisting>
+SELECT c.oid::regclass as table_name,
+greatest(age(c.relfrozenxid),
+ age(t.relfrozenxid)) as xid_age,
+mxid_age(c.relminmxid)
+FROM pg_class c
+LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
+WHERE c.relkind IN ('r', 'm');
+
+SELECT datname,
+age(datfrozenxid) as xid_age,
+mxid_age(datminmxid)
+FROM pg_database;
+</programlisting>
+
+ The <function>age</function> function returns the number of
+ transactions from <structfield>relfrozenxid</structfield> to the
+ next unallocated transaction ID. The
+ <function>mxid_age</function> function the number of MultiXact
+ IDs from <structfield>relminmxid</structfield> to the next
+ unallocated MultiXact ID.
+ </para>
+
+ <para>
+ The system should always have significant XID allocation slack
+ capacity. Ideally, the greatest
+ <literal>age(relfrozenxid)</literal>/<literal>age(datfrozenxid)</literal>
+ in the system will never be more than a fraction of the 2.1
+ billion XID hard limit described in <xref
+ linkend="aggressive-vacuum"/>. The default
+ <varname>vacuum_freeze_table_age</varname> setting of 200 million
+ transactions implies that the system should never use
+ significantly more than about 10% of that hard limit.
+ </para>
+
+ <para>
+ There is little advantage in routinely allowing the greatest
+ <literal>age(relfrozenxid)</literal> in the system to get
+ anywhere near to the 2.1 billion XID hard limit. Putting off the
+ work of freezing can only reduce the absolute amount of
+ <acronym>WAL</acronym> written by <command>VACUUM</command> when
+ <command>VACUUM</command> thereby completely avoids freezing rows
+ that are deleted before long anyway. There is little or no
+ disadvantage from lowering <varname>vacuum_freeze_table_age</varname>
+ to make aggressive <command>VACUUM</command>s more frequent, at
+ least in tables where newly frozen pages almost always remain
+ all-frozen forever. Note also that anything that leads to
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> advancing less frequently
+ (such as a higher <varname>vacuum_freeze_table_age</varname>
+ setting) will also increase the on-disk space required to store
+ additional transaction status information, as described in <xref
+ linkend="vacuum-truncate-xact-status"/>.
+ </para>
+
+ </sect3>
+
+ <sect3 id="xid-stop-limit">
+ <title><literal>xidStopLimit</literal> mode</title>
+ <para>
+ If for some reason autovacuum utterly fails to advance any
+ table's <structfield>relfrozenxid</structfield> or
+ <structfield>relminmxid</structfield> for an extended period, and
+ if XIDs and/or MultiXact IDs continue to be allocated, the system
+ will begin to emit warning messages like this when the database's
+ oldest XIDs reach forty million transactions from the 2.1 billion
+ XID hard limit described in <xref linkend="aggressive-vacuum"/>:
<programlisting>
WARNING: database "mydb" must be vacuumed within 39985967 transactions
HINT: To avoid a database shutdown, execute a database-wide VACUUM in that database.
</programlisting>
- (A manual <command>VACUUM</command> should fix the problem, as suggested by the
- hint; but note that the <command>VACUUM</command> must be performed by a
- superuser, else it will fail to process system catalogs and thus not
- be able to advance the database's <structfield>datfrozenxid</structfield>.)
- If these warnings are
- ignored, the system will shut down and refuse to start any new
- transactions once there are fewer than three million transactions left
- until wraparound:
+ (A manual <command>VACUUM</command> should fix the problem, as suggested by the
+ hint; but note that the <command>VACUUM</command> must be performed by a
+ superuser, else it will fail to process system catalogs and thus not
+ be able to advance the database's <structfield>datfrozenxid</structfield>.)
+ If these warnings are ignored, the system will eventually refuse
+ to start any new transactions. This happens at the point that
+ there are fewer than three million transactions left:
<programlisting>
ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb"
HINT: Stop the postmaster and vacuum that database in single-user mode.
</programlisting>
- The three-million-transaction safety margin exists to let the
- administrator recover without data loss, by manually executing the
- required <command>VACUUM</command> commands. However, since the system will not
- execute commands once it has gone into the safety shutdown mode,
- the only way to do this is to stop the server and start the server in single-user
- mode to execute <command>VACUUM</command>. The shutdown mode is not enforced
- in single-user mode. See the <xref linkend="app-postgres"/> reference
- page for details about using single-user mode.
- </para>
-
- <sect3 id="vacuum-for-multixact-wraparound">
- <title>Multixacts and Wraparound</title>
-
- <indexterm>
- <primary>MultiXactId</primary>
- </indexterm>
-
- <indexterm>
- <primary>wraparound</primary>
- <secondary>of multixact IDs</secondary>
- </indexterm>
-
- <para>
- <firstterm>Multixact IDs</firstterm> are used to support row locking by
- multiple transactions. Since there is only limited space in a tuple
- header to store lock information, that information is encoded as
- a <quote>multiple transaction ID</quote>, or multixact ID for short,
- whenever there is more than one transaction concurrently locking a
- row. Information about which transaction IDs are included in any
- particular multixact ID is stored separately in
- the <filename>pg_multixact</filename> subdirectory, and only the multixact ID
- appears in the <structfield>xmax</structfield> field in the tuple header.
- Like transaction IDs, multixact IDs are implemented as a
- 32-bit counter and corresponding storage, all of which requires
- careful aging management, storage cleanup, and wraparound handling.
- There is a separate storage area which holds the list of members in
- each multixact, which also uses a 32-bit counter and which must also
- be managed.
+ The three-million-transaction safety margin exists to let the
+ administrator recover without data loss, by manually executing the
+ required <command>VACUUM</command> commands. However, since the system will not
+ execute commands once it has gone into the safety shutdown mode,
+ the only way to do this is to stop the server and start the server in single-user
+ mode to execute <command>VACUUM</command>. The shutdown mode is not enforced
+ in single-user mode. See the <xref linkend="app-postgres"/> reference
+ page for details about using single-user mode.
</para>
<para>
- Whenever <command>VACUUM</command> scans any part of a table, it will replace
- any multixact ID it encounters which is older than
- <xref linkend="guc-vacuum-multixact-freeze-min-age"/>
- by a different value, which can be the zero value, a single
- transaction ID, or a newer multixact ID. For each table,
- <structname>pg_class</structname>.<structfield>relminmxid</structfield> stores the oldest
- possible multixact ID still appearing in any tuple of that table.
- If this value is older than
- <xref linkend="guc-vacuum-multixact-freeze-table-age"/>, an aggressive
- vacuum is forced. As discussed in the previous section, an aggressive
- vacuum means that only those pages which are known to be all-frozen will
- be skipped. <function>mxid_age()</function> can be used on
- <structname>pg_class</structname>.<structfield>relminmxid</structfield> to find its age.
- </para>
-
- <para>
- Aggressive <command>VACUUM</command>s, regardless of what causes
- them, are <emphasis>guaranteed</emphasis> to be able to advance
- the table's <structfield>relminmxid</structfield>.
- Eventually, as all tables in all databases are scanned and their
- oldest multixact values are advanced, on-disk storage for older
- multixacts can be removed.
- </para>
-
- <para>
- As a safety device, an aggressive vacuum scan will
- occur for any table whose multixact-age is greater than <xref
- linkend="guc-autovacuum-multixact-freeze-max-age"/>. Also, if the
- storage occupied by multixacts members exceeds 2GB, aggressive vacuum
- scans will occur more often for all tables, starting with those that
- have the oldest multixact-age. Both of these kinds of aggressive
- scans will occur even if autovacuum is nominally disabled.
+ In emergencies, <command>VACUUM</command> will take extraordinary
+ measures to avoid <literal>xidStopLimit</literal> mode. A
+ failsafe mechanism is triggered when the table's
+ <structfield>relfrozenxid</structfield> attains an age of <xref
+ linkend="guc-vacuum-failsafe-age"/> XIDs, or when the table's
+ <structfield>relminmxid</structfield> attains an age of <xref
+ linkend="guc-vacuum-multixact-failsafe-age"/> MultiXact IDs.
+ The failsafe prioritizes advancing
+ <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield> as quickly as possible.
+ Once the failsafe triggers, <command>VACUUM</command> bypasses
+ all remaining non-essential maintenance tasks, and stops applying
+ any cost-based delay that was in effect. Any <glossterm
+ linkend="glossary-buffer-access-strategy">Buffer Access
+ Strategy</glossterm> in use will also be disabled.
</para>
</sect3>
</sect2>
@@ -787,12 +1156,23 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
<title>Updating the Visibility Map</title>
<para>
- Vacuum maintains a <link linkend="storage-vm">visibility
- map</link> for each table to keep track of which pages contain
- only tuples that are known to be visible to all active
- transactions (and all future transactions, until the page is again
- modified). This has two purposes. First, vacuum itself can skip
- such pages on the next run, since there is nothing to clean up.
+ <command>VACUUM</command> maintains a <link
+ linkend="storage-vm">visibility map</link> for each table to keep
+ track of which pages contain only tuples that are known to be
+ visible to all active transactions (and all future transactions,
+ at least until the page is modified). A separate bit tracks
+ whether all of the tuples are frozen.
+ </para>
+
+ <para>
+ The visibility map serves two purposes.
+ </para>
+
+ <para>
+ First, <command>VACUUM</command> itself can skip such pages on the
+ next run, since there is nothing to clean up. Even <link
+ linkend="aggressive-vacuum">aggressive <command>VACUUM</command>s</link>
+ can skip pages that are both all-visible and all-frozen.
</para>
<para>
@@ -812,6 +1192,65 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect2>
+ <sect2 id="vacuum-truncate-xact-status">
+ <title>Truncating transaction status information</title>
+
+ <para>
+ Anything that influences when and how
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> advance will also directly
+ affect the high watermark storage overhead needed to store
+ historical transaction status information. For example,
+ increasing <varname>autovacuum_freeze_max_age</varname> (and
+ <varname>vacuum_freeze_table_age</varname> along with it) will
+ make the <filename>pg_xact</filename> and
+ <filename>pg_commit_ts</filename> subdirectories of the database
+ cluster take more space, because they store the commit/abort
+ status and (if <varname>track_commit_timestamp</varname> is enabled)
+ timestamp of all transactions back to the
+ <varname>datfrozenxid</varname> horizon (the earliest
+ <varname>datfrozenxid</varname> among all databases in the
+ cluster).
+ </para>
+
+ <para>
+ The commit status uses two bits per transaction. The default
+ <varname>autovacuum_freeze_max_age</varname> setting of 200
+ million transactions translates to about 50MB of
+ <filename>pg_xact</filename> storage. When
+ <varname>track_commit_timestamp</varname> is enabled, about 2GB of
+ <filename>pg_commit_ts</filename> storage will also be required.
+ </para>
+
+ <para>
+ MultiXact ID status information storage uses two separate
+ underlying <acronym>SLRU</acronym> storage areas:
+ <filename>pg_multixact/members</filename>, and
+ <filename>pg_multixact/offsets</filename>. There is no simple
+ formula to determine the storage overhead per MultiXact ID, since
+ in general MultiXact IDs have a variable number of member XIDs.
+ Note, however, that if <filename>pg_multixact/members</filename>
+ exceeds 2GB, then the effective value of
+ <varname>autovacuum_multixact_freeze_max_age</varname> used by
+ <command>VACUUM</command> will be lower, resulting in more
+ frequent aggressive mode <command>VACUUM</command>s.
+ </para>
+
+ <para>
+ Truncation of transaction status information is only possible at
+ the end of <command>VACUUM</command>s that advance the earliest
+ <structfield>relfrozenxid</structfield> (in the case of
+ <filename>pg_xact</filename> and
+ <filename>pg_commit_ts</filename>), or the earliest
+ <structfield>relminmxid</structfield> (in the case of
+ <filename>pg_multixact/members</filename> and
+ <filename>pg_multixact/offsets</filename>) among all tables in the
+ entire database (assuming that its the database with the earliest
+ <structfield>datfrozenxid</structfield> and
+ <structfield>datminmxid</structfield> in the entire cluster).
+ </para>
+ </sect2>
+
<sect2 id="vacuum-for-statistics">
<title>Updating Planner Statistics</title>
@@ -927,7 +1366,7 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</tip>
</sect2>
-</sect1>
+ </sect1>
<sect1 id="routine-reindex">
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 10ef699fa..8aa332fcf 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1515,7 +1515,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
and/or <command>ANALYZE</command> operations on this table following the rules
discussed in <xref linkend="autovacuum"/>.
If false, this table will not be autovacuumed, except to prevent
- transaction ID wraparound. See <xref linkend="vacuum-for-wraparound"/> for
+ transaction ID wraparound. See <xref linkend="freezing-xid-space"/> for
more about wraparound prevention.
Note that the autovacuum daemon does not run at all (except to prevent
transaction ID wraparound) if the <xref linkend="guc-autovacuum"/>
diff --git a/doc/src/sgml/ref/prepare_transaction.sgml b/doc/src/sgml/ref/prepare_transaction.sgml
index f4f6118ac..ede50d6f7 100644
--- a/doc/src/sgml/ref/prepare_transaction.sgml
+++ b/doc/src/sgml/ref/prepare_transaction.sgml
@@ -128,7 +128,7 @@ PREPARE TRANSACTION <replaceable class="parameter">transaction_id</replaceable>
This will interfere with the ability of <command>VACUUM</command> to reclaim
storage, and in extreme cases could cause the database to shut down
to prevent transaction ID wraparound (see <xref
- linkend="vacuum-for-wraparound"/>). Keep in mind also that the transaction
+ linkend="freezing-xid-space"/>). Keep in mind also that the transaction
continues to hold whatever locks it held. The intended usage of the
feature is that a prepared transaction will normally be committed or
rolled back as soon as an external transaction manager has verified that
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index 57bc4c23e..95efe7d36 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -123,7 +123,9 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
<term><literal>FREEZE</literal></term>
<listitem>
<para>
- Selects aggressive <quote>freezing</quote> of tuples.
+ Makes <quote>freezing</quote> <emphasis>maximally</emphasis>
+ aggressive, and forces <command>VACUUM</command> to use its
+ <link linkend="aggressive-vacuum">aggressive strategy</link>.
Specifying <literal>FREEZE</literal> is equivalent to performing
<command>VACUUM</command> with the
<xref linkend="guc-vacuum-freeze-min-age"/> and
@@ -219,7 +221,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
there are many dead tuples in the table. This may be useful
when it is necessary to make <command>VACUUM</command> run as
quickly as possible to avoid imminent transaction ID wraparound
- (see <xref linkend="vacuum-for-wraparound"/>). However, the
+ (see <xref linkend="freezing-xid-space"/>). However, the
wraparound failsafe mechanism controlled by <xref
linkend="guc-vacuum-failsafe-age"/> will generally trigger
automatically to avoid transaction ID wraparound failure, and
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index da2393783..b61d523c2 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -233,7 +233,7 @@ PostgreSQL documentation
ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
This setting is useful for prioritizing tables to process to prevent
multixact ID wraparound (see
- <xref linkend="vacuum-for-multixact-wraparound"/>).
+ <xref linkend="freezing-xid-space"/>).
</para>
<para>
For the purposes of this option, the multixact ID age of a relation is
@@ -254,7 +254,7 @@ PostgreSQL documentation
transaction ID age of at least
<replaceable class="parameter">xid_age</replaceable>. This setting
is useful for prioritizing tables to process to prevent transaction
- ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ ID wraparound (see <xref linkend="freezing-xid-space"/>).
</para>
<para>
For the purposes of this option, the transaction ID age of a relation
diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml
index 0762442e1..e372a7875 100644
--- a/doc/src/sgml/xact.sgml
+++ b/doc/src/sgml/xact.sgml
@@ -185,7 +185,7 @@
rows and can be inspected using the <xref linkend="pgrowlocks"/>
extension. Row-level read locks might also require the assignment
of multixact IDs (<literal>mxid</literal>; see <xref
- linkend="vacuum-for-multixact-wraparound"/>).
+ linkend="freezing-xid-space"/>).
</para>
</sect1>
--
2.40.1
[application/octet-stream] v3-0008-Overhaul-Recovering-Disk-Space-vacuuming-docs.patch (11.5K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/5-v3-0008-Overhaul-Recovering-Disk-Space-vacuuming-docs.patch)
download | inline diff:
From 7de5ab5e2389a6f5dc41f0b9e23e8a169db47834 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 12:33:42 -0700
Subject: [PATCH v3 8/9] Overhaul "Recovering Disk Space" vacuuming docs.
Say a lot more about the possible impact of long-running transactions on
VACUUM. Remove all talk of administrators getting by without
autovacuum; at most administrators might want to schedule manual VACUUM
operations to supplement autovacuum (this documentation was written at a
time when the visibility map didn't exist, even in its most basic form).
Also describe VACUUM FULL as an entirely different kind of operation to
conventional lazy vacuum.
XXX Open question for this commit:
I wonder if it would make sense to move all of that stuff into its own
new sect1 of "Chapter 29. Monitoring Disk Usage" -- something along
the lines of "what to do about bloat when all else fails, when the
problem gets completely out of hand". Naturally we'd link to this new
section from "Routine Vacuuming".
XXX For now, a lot of the information about CLUSTER and VACUUM FULL is
moved into Note/Warning boxes. This arrangement is definitely going to
be temporary.
---
doc/src/sgml/maintenance.sgml | 174 +++++++++++++++++++---------------
1 file changed, 96 insertions(+), 78 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 36f481aba..5546d8c7d 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -369,100 +369,118 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
This approach is necessary to gain the benefits of multiversion
concurrency control (<acronym>MVCC</acronym>, see <xref linkend="mvcc"/>): the row version
must not be deleted while it is still potentially visible to other
- transactions. But eventually, an outdated or deleted row version is no
- longer of interest to any transaction. The space it occupies must then be
- reclaimed for reuse by new rows, to avoid unbounded growth of disk
- space requirements. This is done by running <command>VACUUM</command>.
+ transactions. A deleted row version (whether from an
+ <command>UPDATE</command> or <command>DELETE</command>) will
+ usually cease to be of interest to any still-running transaction
+ shortly after the original deleting transaction commits.
</para>
<para>
- The standard form of <command>VACUUM</command> removes dead row
- versions in tables and indexes and marks the space available for
- future reuse. However, it will not return the space to the operating
- system, except in the special case where one or more pages at the
- end of a table become entirely free and an exclusive table lock can be
- easily obtained. In contrast, <command>VACUUM FULL</command> actively compacts
- tables by writing a complete new version of the table file with no dead
- space. This minimizes the size of the table, but can take a long time.
- It also requires extra disk space for the new copy of the table, until
- the operation completes.
+ The space dead tuples occupy must eventually be reclaimed for
+ reuse by new rows, to avoid unbounded growth of disk space
+ requirements. Reclaiming space from dead rows is
+ <command>VACUUM</command>'s main responsibility.
</para>
<para>
- The usual goal of routine vacuuming is to do standard <command>VACUUM</command>s
- often enough to avoid needing <command>VACUUM FULL</command>. The
- autovacuum daemon attempts to work this way, and in fact will
- never issue <command>VACUUM FULL</command>. In this approach, the idea
- is not to keep tables at their minimum size, but to maintain steady-state
- usage of disk space: each table occupies space equivalent to its
- minimum size plus however much space gets used up between vacuum runs.
- Although <command>VACUUM FULL</command> can be used to shrink a table back
- to its minimum size and return the disk space to the operating system,
- there is not much point in this if the table will just grow again in the
- future. Thus, moderately-frequent standard <command>VACUUM</command> runs are a
- better approach than infrequent <command>VACUUM FULL</command> runs for
- maintaining heavily-updated tables.
+ The <glossterm linkend="glossary-xid">transaction ID number
+ (<acronym>XID</acronym>)</glossterm> based cutoff point that
+ <command>VACUUM</command> uses to determine if a deleted tuple is
+ safe to physically remove is reported under <literal>removable
+ cutoff</literal> in the server log when autovacuum logging
+ (controlled by <xref linkend="guc-log-autovacuum-min-duration"/>)
+ reports on a <command>VACUUM</command> operation executed by
+ autovacuum. Tuples that are not yet safe to remove are counted as
+ <literal>dead but not yet removable</literal> tuples in the log
+ report. <command>VACUUM</command> establishes its
+ <literal>removable cutoff</literal> once, at the start of the
+ operation. Any older <acronym>MVCC</acronym> snapshot (or
+ transaction that allocates an XID) that's still running when the
+ cutoff is established may hold it back.
</para>
- <para>
- Some administrators prefer to schedule vacuuming themselves, for example
- doing all the work at night when load is low.
- The difficulty with doing vacuuming according to a fixed schedule
- is that if a table has an unexpected spike in update activity, it may
- get bloated to the point that <command>VACUUM FULL</command> is really necessary
- to reclaim space. Using the autovacuum daemon alleviates this problem,
- since the daemon schedules vacuuming dynamically in response to update
- activity. It is unwise to disable the daemon completely unless you
- have an extremely predictable workload. One possible compromise is
- to set the daemon's parameters so that it will only react to unusually
- heavy update activity, thus keeping things from getting out of hand,
- while scheduled <command>VACUUM</command>s are expected to do the bulk of the
- work when the load is typical.
- </para>
+ <caution>
+ <para>
+ It's important that no long-running transactions ever be allowed
+ to hold back every <command>VACUUM</command> operation's cutoff
+ for an extended period. You may wish to add monitoring to alert
+ on this.
+ </para>
+ </caution>
+
+ <note>
+ <para>
+ <command>VACUUM</command> can remove tuples inserted by aborted
+ transactions immediately
+ </para>
+ </note>
<para>
- For those not using autovacuum, a typical approach is to schedule a
- database-wide <command>VACUUM</command> once a day during a low-usage period,
- supplemented by more frequent vacuuming of heavily-updated tables as
- necessary. (Some installations with extremely high update rates vacuum
- their busiest tables as often as once every few minutes.) If you have
- multiple databases in a cluster, don't forget to
- <command>VACUUM</command> each one; the program <xref
- linkend="app-vacuumdb"/> might be helpful.
+ <command>VACUUM</command> usually won't return space to the
+ operating system. There is one exception: space is returned to the
+ OS whenever a group of contiguous pages appears at the end of a
+ table. <command>VACUUM</command> must acquire an <literal>ACCESS
+ EXCLUSIVE</literal> lock to perform relation truncation. You can
+ disable relation truncation by setting the table's
+ <varname>vacuum_truncate</varname> storage parameter to
+ <literal>off</literal>.
</para>
<tip>
- <para>
- Plain <command>VACUUM</command> may not be satisfactory when
- a table contains large numbers of dead row versions as a result of
- massive update or delete activity. If you have such a table and
- you need to reclaim the excess disk space it occupies, you will need
- to use <command>VACUUM FULL</command>, or alternatively
- <link linkend="sql-cluster"><command>CLUSTER</command></link>
- or one of the table-rewriting variants of
- <link linkend="sql-altertable"><command>ALTER TABLE</command></link>.
- These commands rewrite an entire new copy of the table and build
- new indexes for it. All these options require an
- <literal>ACCESS EXCLUSIVE</literal> lock. Note that
- they also temporarily use extra disk space approximately equal to the size
- of the table, since the old copies of the table and indexes can't be
- released until the new ones are complete.
- </para>
+ <para>
+ If you have a table whose entire contents are deleted on a
+ periodic basis, consider doing it with <link
+ linkend="sql-truncate"><command>TRUNCATE</command></link> rather
+ than relying on <command>VACUUM</command>.
+ <command>TRUNCATE</command> removes the entire contents of the
+ table immediately, avoiding the need to set
+ <structfield>xmax</structfield> to the deleting transaction's XID.
+ One disadvantage is that strict MVCC semantics are violated.
+ </para>
</tip>
-
<tip>
- <para>
- If you have a table whose entire contents are deleted on a periodic
- basis, consider doing it with
- <link linkend="sql-truncate"><command>TRUNCATE</command></link> rather
- than using <command>DELETE</command> followed by
- <command>VACUUM</command>. <command>TRUNCATE</command> removes the
- entire content of the table immediately, without requiring a
- subsequent <command>VACUUM</command> or <command>VACUUM
- FULL</command> to reclaim the now-unused disk space.
- The disadvantage is that strict MVCC semantics are violated.
- </para>
+ <para>
+ <command>VACUUM FULL</command> or <command>CLUSTER</command> can
+ be useful when dealing with extreme amounts of dead tuples. It
+ can reclaim more disk space, but runs much more slowly. It
+ rewrites an entire new copy of the table and rebuilds all of the
+ table's indexes. As a result, <command>VACUUM FULL</command> and
+ <command>CLUSTER</command> typically have a much higher overhead
+ than <command>VACUUM</command>. Generally, therefore,
+ administrators should avoid using <command>VACUUM FULL</command>
+ except in the most extreme cases.
+ </para>
</tip>
+ <note>
+ <para>
+ Although <command>VACUUM FULL</command> is technically an option
+ of the <command>VACUUM</command> command, <command>VACUUM
+ FULL</command> uses a completely different implementation.
+ <command>VACUUM FULL</command> is essentially a variant of
+ <command>CLUSTER</command>. (The name <command>VACUUM
+ FULL</command> is historical; the original implementation was
+ somewhat closer to standard <command>VACUUM</command>.)
+ </para>
+ </note>
+ <warning>
+ <para>
+ <command>TRUNCATE</command>, <command>VACUUM FULL</command>, and
+ <command>CLUSTER</command> all require an <literal>ACCESS
+ EXCLUSIVE</literal> lock, which can be highly disruptive
+ (<command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command>, and <command>DELETE</command> commands
+ won't be able to run at the same time).
+ </para>
+ </warning>
+ <warning>
+ <para>
+ <command>VACUUM FULL</command> and <command>CLUSTER</command>
+ temporarily use extra disk space. The extra space required is
+ approximately equal to the size of the table, since the old
+ copies of the table and indexes can't be released until the new
+ ones are complete.
+ </para>
+ </warning>
</sect2>
<sect2 id="vacuum-for-wraparound">
--
2.40.1
[application/octet-stream] v3-0007-Make-maintenance.sgml-more-autovacuum-orientated.patch (8.6K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/6-v3-0007-Make-maintenance.sgml-more-autovacuum-orientated.patch)
download | inline diff:
From b1bba6df919d98545ee4d49f6bb8b4892ed55b27 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 12:11:10 -0700
Subject: [PATCH v3 7/9] Make maintenance.sgml more autovacuum-orientated.
Now that it's no longer in its own sect2, shorten the "Vacuuming basics"
content, and make it more autovacuum-orientated. This gives much less
prominence to VACUUM FULL, which has little place in a section about
autovacuum. We no longer define avoiding the need to run VACUUM FULL as
the purpose of vacuuming.
A later commit that overhauls "Recovering Disk Space" will add back a
passing mention of things like VACUUM FULL and TRUNCATE, but only as
something that might be relevant in extreme cases. (Use of these
commands is hopefully neither "Routine" nor "Basic" to most users).
Also add some introductory information about the audience and goals of
the "Routine Vacuuming" section of the docs.
---
doc/src/sgml/maintenance.sgml | 122 +++++++++++++++++++++-------------
1 file changed, 76 insertions(+), 46 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index a05e880fc..36f481aba 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -32,11 +32,12 @@
</para>
<para>
- The other main category of maintenance task is periodic <quote>vacuuming</quote>
- of the database. This activity is discussed in
- <xref linkend="routine-vacuuming"/>. Closely related to this is updating
- the statistics that will be used by the query planner, as discussed in
- <xref linkend="vacuum-for-statistics"/>.
+ The other main category of maintenance task is periodic
+ <quote><link linkend="routine-vacuuming">vacuuming</link></quote> of
+ the database by autovacuum. Configuring autovacuum scheduling is
+ discussed in <xref linkend="autovacuum"/>. Autovacuum also updates
+ the statistics that will be used by the query planner, as discussed
+ in <xref linkend="vacuum-for-statistics"/>.
</para>
<para>
@@ -244,7 +245,7 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</sect1>
<sect1 id="routine-vacuuming">
- <title>Routine Vacuuming</title>
+ <title>Autovacuum Maintenance Tasks</title>
<indexterm zone="routine-vacuuming">
<primary>vacuum</primary>
@@ -252,24 +253,18 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<para>
<productname>PostgreSQL</productname> databases require periodic
- maintenance known as <firstterm>vacuuming</firstterm>. For many installations, it
- is sufficient to let vacuuming be performed by the <firstterm>autovacuum
- daemon</firstterm>, which is described in <xref linkend="autovacuum"/>. You might
- need to adjust the autovacuuming parameters described there to obtain best
- results for your situation. Some database administrators will want to
- supplement or replace the daemon's activities with manually-managed
- <command>VACUUM</command> commands, which typically are executed according to a
- schedule by <application>cron</application> or <application>Task
- Scheduler</application> scripts. To set up manually-managed vacuuming properly,
- it is essential to understand the issues discussed in the next few
- subsections. Administrators who rely on autovacuuming may still wish
- to skim this material to help them understand and adjust autovacuuming.
+ maintenance known as <firstterm>vacuuming</firstterm>, and require
+ periodic updates to the statistics used by the
+ <productname>PostgreSQL</productname> query planner. The <link
+ linkend="sql-vacuum"><command>VACUUM</command></link> and <link
+ linkend="sql-analyze"><command>ANALYZE</command></link> commands
+ perform these maintenance tasks. The <firstterm>autovacuum
+ daemon</firstterm> automatically schedules execution of
+ maintenance, based on the requirements of the workload.
</para>
-
<para>
- <productname>PostgreSQL</productname>'s
- <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
- process each table on a regular basis for several reasons:
+ The autovacuum daemon has to process each table regularly for
+ several reasons:
<orderedlist>
<listitem>
@@ -295,34 +290,69 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
</orderedlist>
- Each of these reasons dictates performing <command>VACUUM</command> operations
- of varying frequency and scope, as explained in the following subsections.
+ Maintenance work within the scope of items 1, 2, 3, and 4 is
+ performed by the <command>VACUUM</command> command internally. The
+ <command>ANALYZE</command> command handles maintenance work within
+ the scope of item 5 (maintenance of planner statistics) internally.
</para>
-
<para>
- There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
- and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
- disk space but runs much more slowly. Also,
- the standard form of <command>VACUUM</command> can run in parallel with production
- database operations. (Commands such as <command>SELECT</command>,
- <command>INSERT</command>, <command>UPDATE</command>, and
- <command>DELETE</command> will continue to function normally, though you
- will not be able to modify the definition of a table with commands such as
- <command>ALTER TABLE</command> while it is being vacuumed.)
- <command>VACUUM FULL</command> requires an
- <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
- working on, and therefore cannot be done in parallel with other use
- of the table. Generally, therefore,
- administrators should strive to use standard <command>VACUUM</command> and
- avoid <command>VACUUM FULL</command>.
+ Generally speaking, database administrators that are new to tuning
+ autovacuum should start by considering adjusting autovacuum's
+ scheduling. Autovacuum scheduling is controlled via threshold
+ settings. These settings determine when autovacuum should launch a
+ worker to run <command>VACUUM</command> and/or
+ <command>ANALYZE</command>; see the previous section, <xref
+ linkend="autovacuum"/>. This section provides additional
+ information about the design and goals of autovacuum,
+ <command>VACUUM</command>, and <command>ANALYZE</command>. The
+ intended audience is database administrators that wish to perform
+ more advanced tuning of autovacuum, with any of the following goals
+ in mind:
</para>
-
+ <itemizedlist>
+ <listitem>
+ <para>
+ Tuning <command>VACUUM</command> to improve query response times.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Making sure that <command>VACUUM</command>'s management of the
+ Transaction ID address space is operating normally.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Tuning <command>VACUUM</command> for performance stability.
+ </para>
+ </listitem>
+ </itemizedlist>
<para>
- <command>VACUUM</command> creates a substantial amount of I/O
- traffic, which can cause poor performance for other active sessions.
- There are configuration parameters that can be adjusted to reduce the
- performance impact of background vacuuming — see
- <xref linkend="runtime-config-resource-vacuum-cost"/>.
+ With larger installations, tuning autovacuum usually won't be a
+ once-off task; it is best to approach tuning as an iterative,
+ applied process. FIXME Expand this to describe the intended
+ audience on goals in a fully worked out way.
+ </para>
+ <para>
+ Autovacuum creates a substantial amount of I/O traffic, which can
+ cause poor performance for other active sessions. There are
+ configuration parameters that you can adjust to reduce the
+ performance impact of background vacuuming. See the
+ autovacuum-specific cost delay settings described in <xref
+ linkend="runtime-config-autovacuum"/>, and additional cost delay
+ settings described in <xref
+ linkend="runtime-config-resource-vacuum-cost"/>.
+ </para>
+ <para>
+ Some database administrators will want to supplement the daemon's
+ activities with manually-managed <command>VACUUM</command>
+ commands. Scripting tools like <application>cron</application> and
+ <application>Task Manager</application> can be of help with this.
+ It can be useful to perform off-hours <command>VACUUM</command>
+ commands during periods where reduced load is expected. Almost all
+ of the contents of this section apply equally to manually-issued
+ <command>VACUUM</command> and <command>ANALYZE</command>
+ operations.
</para>
<sect2 id="vacuum-for-space-recovery">
--
2.40.1
[application/octet-stream] v3-0006-Merge-basic-vacuuming-sect2-into-sect1-introducti.patch (6.2K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/7-v3-0006-Merge-basic-vacuuming-sect2-into-sect1-introducti.patch)
download | inline diff:
From 4ed889f2452e832028bc198271b2ba5b7856536c Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 11:44:45 -0700
Subject: [PATCH v3 6/9] Merge "basic vacuuming" sect2 into sect1 introduction.
This doesn't change any of the content itself. It just merges the
original text into the sect1 text that immediately preceded it.
This is preparation for the next commit, which will remove most of the
text "relocated" in this commit. This structure should make things a
little easier for doc translators.
This commit is the last one that could be considered mechanical
restructuring/refactoring of existing text.
---
doc/src/sgml/maintenance.sgml | 106 ++++++++++++++++------------------
1 file changed, 51 insertions(+), 55 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index cb6f28e1e..a05e880fc 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -266,68 +266,64 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
to skim this material to help them understand and adjust autovacuuming.
</para>
- <sect2 id="vacuum-basics">
- <title>Vacuuming Basics</title>
+ <para>
+ <productname>PostgreSQL</productname>'s
+ <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
+ process each table on a regular basis for several reasons:
- <para>
- <productname>PostgreSQL</productname>'s
- <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
- process each table on a regular basis for several reasons:
+ <orderedlist>
+ <listitem>
+ <simpara>To recover or reuse disk space occupied by updated or deleted
+ rows.</simpara>
+ </listitem>
- <orderedlist>
- <listitem>
- <simpara>To recover or reuse disk space occupied by updated or deleted
- rows.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To protect against loss of very old data due to
+ <firstterm>transaction ID wraparound</firstterm> or
+ <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ </listitem>
- <listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To update the visibility map, which speeds
+ up <link linkend="indexes-index-only-scans">index-only
+ scans</link>.</simpara>
+ </listitem>
- <listitem>
- <simpara>To update the visibility map, which speeds
- up <link linkend="indexes-index-only-scans">index-only
- scans</link>.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To update data statistics used by the
+ <productname>PostgreSQL</productname> query planner.</simpara>
+ </listitem>
+ </orderedlist>
- <listitem>
- <simpara>To update data statistics used by the
- <productname>PostgreSQL</productname> query planner.</simpara>
- </listitem>
- </orderedlist>
+ Each of these reasons dictates performing <command>VACUUM</command> operations
+ of varying frequency and scope, as explained in the following subsections.
+ </para>
- Each of these reasons dictates performing <command>VACUUM</command> operations
- of varying frequency and scope, as explained in the following subsections.
- </para>
+ <para>
+ There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
+ and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
+ disk space but runs much more slowly. Also,
+ the standard form of <command>VACUUM</command> can run in parallel with production
+ database operations. (Commands such as <command>SELECT</command>,
+ <command>INSERT</command>, <command>UPDATE</command>, and
+ <command>DELETE</command> will continue to function normally, though you
+ will not be able to modify the definition of a table with commands such as
+ <command>ALTER TABLE</command> while it is being vacuumed.)
+ <command>VACUUM FULL</command> requires an
+ <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
+ working on, and therefore cannot be done in parallel with other use
+ of the table. Generally, therefore,
+ administrators should strive to use standard <command>VACUUM</command> and
+ avoid <command>VACUUM FULL</command>.
+ </para>
- <para>
- There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
- and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
- disk space but runs much more slowly. Also,
- the standard form of <command>VACUUM</command> can run in parallel with production
- database operations. (Commands such as <command>SELECT</command>,
- <command>INSERT</command>, <command>UPDATE</command>, and
- <command>DELETE</command> will continue to function normally, though you
- will not be able to modify the definition of a table with commands such as
- <command>ALTER TABLE</command> while it is being vacuumed.)
- <command>VACUUM FULL</command> requires an
- <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
- working on, and therefore cannot be done in parallel with other use
- of the table. Generally, therefore,
- administrators should strive to use standard <command>VACUUM</command> and
- avoid <command>VACUUM FULL</command>.
- </para>
-
- <para>
- <command>VACUUM</command> creates a substantial amount of I/O
- traffic, which can cause poor performance for other active sessions.
- There are configuration parameters that can be adjusted to reduce the
- performance impact of background vacuuming — see
- <xref linkend="runtime-config-resource-vacuum-cost"/>.
- </para>
- </sect2>
+ <para>
+ <command>VACUUM</command> creates a substantial amount of I/O
+ traffic, which can cause poor performance for other active sessions.
+ There are configuration parameters that can be adjusted to reduce the
+ performance impact of background vacuuming — see
+ <xref linkend="runtime-config-resource-vacuum-cost"/>.
+ </para>
<sect2 id="vacuum-for-space-recovery">
<title>Recovering Disk Space</title>
--
2.40.1
[application/octet-stream] v3-0005-Move-Interpreting-XID-stamps-from-tuple-headers.patch (12.9K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/8-v3-0005-Move-Interpreting-XID-stamps-from-tuple-headers.patch)
download | inline diff:
From 916e6a6121f03b65f361e1ca064fb5d4b6181cee Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 12:41:00 -0700
Subject: [PATCH v3 5/9] Move Interpreting XID stamps from tuple headers.
This is intended to be fairly close to a mechanical change. It isn't
entirely mechanical, though, since the original wording has been
slightly modified for it to work in context.
Structuring things this way should make life a little easier for doc
translators.
---
doc/src/sgml/maintenance.sgml | 80 ++++------------------
doc/src/sgml/xact.sgml | 125 ++++++++++++++++++++++++++++------
2 files changed, 120 insertions(+), 85 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index e130dfdbd..cb6f28e1e 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -447,75 +447,25 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<secondary>wraparound</secondary>
</indexterm>
- <indexterm>
- <primary>wraparound</primary>
- <secondary>of transaction IDs</secondary>
- </indexterm>
+ <indexterm>
+ <primary>wraparound</primary>
+ <secondary>of transaction IDs</secondary>
+ </indexterm>
<para>
- <productname>PostgreSQL</productname>'s
- <link linkend="mvcc-intro">MVCC</link> transaction semantics
- depend on being able to compare transaction ID (<acronym>XID</acronym>)
- numbers: a row version with an insertion XID greater than the current
- transaction's XID is <quote>in the future</quote> and should not be visible
- to the current transaction. But since transaction IDs have limited size
- (32 bits) a cluster that runs for a long time (more
- than 4 billion transactions) would suffer <firstterm>transaction ID
- wraparound</firstterm>: the XID counter wraps around to zero, and all of a sudden
- transactions that were in the past appear to be in the future — which
- means their output become invisible. In short, catastrophic data loss.
- (Actually the data is still there, but that's cold comfort if you cannot
- get at it.) To avoid this, it is necessary to vacuum every table
- in every database at least once every two billion transactions.
+ <productname>PostgreSQL</productname>'s <link
+ linkend="mvcc-intro">MVCC</link> transaction semantics depend on
+ being able to compare <glossterm linkend="glossary-xid">transaction
+ ID numbers (<acronym>XID</acronym>)</glossterm> to determine
+ whether or not the row is visible to each query's MVCC snapshot
+ (see <xref linkend="interpreting-xid-stamps"/>). But since
+ on-disk storage of transaction IDs in heap pages uses a truncated
+ 32-bit representation to save space (rather than the full 64-bit
+ representation), it is necessary to vacuum every table in every
+ database <emphasis>at least</emphasis> once every two billion
+ transactions (though far more frequent vacuuming is typical).
</para>
- <para>
- The reason that periodic vacuuming solves the problem is that
- <command>VACUUM</command> will mark rows as <emphasis>frozen</emphasis>, indicating that
- they were inserted by a transaction that committed sufficiently far in
- the past that the effects of the inserting transaction are certain to be
- visible to all current and future transactions.
- Normal XIDs are
- compared using modulo-2<superscript>32</superscript> arithmetic. This means
- that for every normal XID, there are two billion XIDs that are
- <quote>older</quote> and two billion that are <quote>newer</quote>; another
- way to say it is that the normal XID space is circular with no
- endpoint. Therefore, once a row version has been created with a particular
- normal XID, the row version will appear to be <quote>in the past</quote> for
- the next two billion transactions, no matter which normal XID we are
- talking about. If the row version still exists after more than two billion
- transactions, it will suddenly appear to be in the future. To
- prevent this, <productname>PostgreSQL</productname> reserves a special XID,
- <literal>FrozenTransactionId</literal>, which does not follow the normal XID
- comparison rules and is always considered older
- than every normal XID.
- Frozen row versions are treated as if the inserting XID were
- <literal>FrozenTransactionId</literal>, so that they will appear to be
- <quote>in the past</quote> to all normal transactions regardless of wraparound
- issues, and so such row versions will be valid until deleted, no matter
- how long that is.
- </para>
-
- <note>
- <para>
- In <productname>PostgreSQL</productname> versions before 9.4, freezing was
- implemented by actually replacing a row's insertion XID
- with <literal>FrozenTransactionId</literal>, which was visible in the
- row's <structname>xmin</structname> system column. Newer versions just set a flag
- bit, preserving the row's original <structname>xmin</structname> for possible
- forensic use. However, rows with <structname>xmin</structname> equal
- to <literal>FrozenTransactionId</literal> (2) may still be found
- in databases <application>pg_upgrade</application>'d from pre-9.4 versions.
- </para>
- <para>
- Also, system catalogs may contain rows with <structname>xmin</structname> equal
- to <literal>BootstrapTransactionId</literal> (1), indicating that they were
- inserted during the first phase of <application>initdb</application>.
- Like <literal>FrozenTransactionId</literal>, this special XID is treated as
- older than every normal XID.
- </para>
- </note>
-
<para>
<xref linkend="guc-vacuum-freeze-min-age"/>
controls how old an XID value has to be before rows bearing that XID will be
diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml
index b467660ee..0762442e1 100644
--- a/doc/src/sgml/xact.sgml
+++ b/doc/src/sgml/xact.sgml
@@ -22,6 +22,8 @@
single-statement transactions.
</para>
+ <sect2 id="virtual-xids">
+ <title>Virtual Transaction IDs</title>
<para>
Every transaction is identified by a unique
<literal>VirtualTransactionId</literal> (also called
@@ -46,29 +48,111 @@
started, particularly if the transaction started with statements that
only performed database reads.
</para>
+ </sect2>
- <para>
- The internal transaction ID type <type>xid</type> is 32 bits wide
- and <link linkend="vacuum-for-wraparound">wraps around</link> every
- 4 billion transactions. A 32-bit epoch is incremented during each
- wraparound. There is also a 64-bit type <type>xid8</type> which
- includes this epoch and therefore does not wrap around during the
- life of an installation; it can be converted to xid by casting.
- The functions in <xref linkend="functions-pg-snapshot"/>
- return <type>xid8</type> values. Xids are used as the
- basis for <productname>PostgreSQL</productname>'s <link
- linkend="mvcc">MVCC</link> concurrency mechanism and streaming
- replication.
- </para>
+ <sect2 id="permanent-xids">
+ <title>Permanent Transaction IDs</title>
+ <para>
+ The internal transaction ID type <type>xid</type> is 32 bits wide
+ and wraps around every
+ 4 billion transactions. A 32-bit epoch is incremented during each
+ wraparound. There is also a 64-bit type <type>xid8</type> which
+ includes this epoch and therefore does not wrap around during the
+ life of an installation; it can be converted to xid by casting.
+ The functions in <xref linkend="functions-pg-snapshot"/>
+ return <type>xid8</type> values. Xids are used as the
+ basis for <productname>PostgreSQL</productname>'s <link
+ linkend="mvcc">MVCC</link> concurrency mechanism and streaming
+ replication.
+ </para>
- <para>
- When a top-level transaction with a (non-virtual) xid commits,
- it is marked as committed in the <filename>pg_xact</filename>
- directory. Additional information is recorded in the
- <filename>pg_commit_ts</filename> directory if <xref
- linkend="guc-track-commit-timestamp"/> is enabled.
- </para>
+ <para>
+ When a top-level transaction with a (non-virtual) xid commits,
+ it is marked as committed in the <filename>pg_xact</filename>
+ directory. Additional information is recorded in the
+ <filename>pg_commit_ts</filename> directory if <xref
+ linkend="guc-track-commit-timestamp"/> is enabled.
+ </para>
+ <sect3 id="interpreting-xid-stamps">
+ <title><type>TransactionId</type> comparison rules</title>
+ <para>
+ The system often needs to compare <structfield>t_xmin</structfield>
+ and <structfield>t_xmax</structfield> fields for MVCC snapshot
+ visibility checks.
+ </para>
+
+ <para>
+ We use a truncated 32-bit representation of transaction IDs, rather
+ than using the full 64-bit representation. The 2.1 billion XIDs
+ <quote>distance</quote> invariant must be preserved because
+ transaction IDs stored in heap row headers use a truncated 32-bit
+ representation (rather than the full 64-bit representation). Since
+ all unfrozen transaction IDs from heap tuple headers
+ <emphasis>must</emphasis> be from the same transaction ID epoch (or
+ from a space in the 64-bit representation that spans two adjoining
+ transaction ID epochs), there isn't any need to store a separate
+ epoch field in each tuple header (see <xref
+ linkend="interpreting-xid-stamps"/> for further details). This
+ scheme has the advantage of requiring much less space than a design
+ that stores an XID epoch alongside each XID stored in each heap
+ tuple header. It has the disadvantage of constraining the system's
+ ability to allocate new XIDs (in the worst case scenario where
+ <literal>xidStopLimit</literal> mode is used to preserve the
+ <quote>distance</quote> invariant).
+ </para>
+
+ <para>
+ <command>VACUUM</command> <link linkend="routine-vacuuming">will
+ mark tuple headers <emphasis>frozen</emphasis></link>, indicating
+ that all eligible rows on the page were inserted by a transaction
+ that committed sufficiently far in the past that the effects of the
+ inserting transaction are certain to be visible to all current and
+ future transactions. Normal XIDs are compared using
+ modulo-2<superscript>32</superscript> arithmetic. This means that
+ for every normal XID, there are two billion XIDs that are
+ <quote>older</quote> and two billion that are <quote>newer</quote>;
+ another way to say it is that the normal XID space is circular with
+ no endpoint. Therefore, once a row version has been created with a
+ particular normal XID, the row version will appear to be <quote>in
+ the past</quote> for the next two billion transactions, no matter
+ which normal XID we are talking about. If the row version still
+ exists after more than two billion transactions, it will suddenly
+ appear to be in the future. To prevent this,
+ <productname>PostgreSQL</productname> reserves a special XID,
+ <literal>FrozenTransactionId</literal>, which does not follow the
+ normal XID comparison rules and is always considered older than
+ every normal XID. Frozen row versions are treated as if the
+ inserting XID were <literal>FrozenTransactionId</literal>, so that
+ they will appear to be <quote>in the past</quote> to all normal
+ transactions regardless of wraparound issues, and so such row
+ versions will be valid until deleted, no matter how long that is.
+ </para>
+
+ <note>
+ <para>
+ In <productname>PostgreSQL</productname> versions before 9.4, freezing was
+ implemented by actually replacing a row's insertion XID
+ with <literal>FrozenTransactionId</literal>, which was visible in the
+ row's <structname>xmin</structname> system column. Newer versions just set a flag
+ bit, preserving the row's original <structname>xmin</structname> for possible
+ forensic use. However, rows with <structname>xmin</structname> equal
+ to <literal>FrozenTransactionId</literal> (2) may still be found
+ in databases <application>pg_upgrade</application>'d from pre-9.4 versions.
+ </para>
+ <para>
+ Also, system catalogs may contain rows with <structname>xmin</structname> equal
+ to <literal>BootstrapTransactionId</literal> (1), indicating that they were
+ inserted during the first phase of <application>initdb</application>.
+ Like <literal>FrozenTransactionId</literal>, this special XID is treated as
+ older than every normal XID.
+ </para>
+ </note>
+ </sect3>
+ </sect2>
+
+ <sect2 id="global-transaction-ids">
+ <title>Global Transaction Identifiers</title>
<para>
In addition to <literal>vxid</literal> and <type>xid</type>,
prepared transactions are also assigned Global Transaction
@@ -77,6 +161,7 @@
prepared transactions. The mapping of GID to xid is shown in <link
linkend="view-pg-prepared-xacts"><structname>pg_prepared_xacts</structname></link>.
</para>
+ </sect2>
</sect1>
<sect1 id="xact-locking">
--
2.40.1
[application/octet-stream] v3-0004-Reorder-routine-vacuuming-sections.patch (16.2K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/9-v3-0004-Reorder-routine-vacuuming-sections.patch)
download | inline diff:
From 6f218d033bb334b5dc42e78cbcf7b33abeafa4f8 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 11:19:50 -0700
Subject: [PATCH v3 4/9] Reorder routine vacuuming sections.
This doesn't change any of the content itself. It is a mechanical
change. The new order flows better because it talks about freezing
directly after talking about space recovery tasks.
Old order:
<sect2 id="vacuum-basics">
<sect2 id="vacuum-for-space-recovery">
<sect2 id="vacuum-for-statistics">
<sect2 id="vacuum-for-visibility-map">
<sect2 id="vacuum-for-wraparound">
New order:
<sect2 id="vacuum-basics">
<sect2 id="vacuum-for-space-recovery">
<sect2 id="vacuum-for-wraparound">
<sect2 id="vacuum-for-visibility-map">
<sect2 id="vacuum-for-statistics">
The new order matches processing order inside vacuumlazy.c. This order
will be easier to work with in two later commits that more or less
rewrite "vacuum-for-wraparound" and "vacuum-for-space-recovery".
(Though it doesn't seem to make the existing content any less meaningful
without the later rewrite commits.)
---
doc/src/sgml/maintenance.sgml | 302 +++++++++++++++++-----------------
1 file changed, 151 insertions(+), 151 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index e8c8647cd..e130dfdbd 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -281,8 +281,9 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To update data statistics used by the
- <productname>PostgreSQL</productname> query planner.</simpara>
+ <simpara>To protect against loss of very old data due to
+ <firstterm>transaction ID wraparound</firstterm> or
+ <firstterm>multixact ID wraparound</firstterm>.</simpara>
</listitem>
<listitem>
@@ -292,9 +293,8 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ <simpara>To update data statistics used by the
+ <productname>PostgreSQL</productname> query planner.</simpara>
</listitem>
</orderedlist>
@@ -439,151 +439,6 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</tip>
</sect2>
- <sect2 id="vacuum-for-statistics">
- <title>Updating Planner Statistics</title>
-
- <indexterm zone="vacuum-for-statistics">
- <primary>statistics</primary>
- <secondary>of the planner</secondary>
- </indexterm>
-
- <indexterm zone="vacuum-for-statistics">
- <primary>ANALYZE</primary>
- </indexterm>
-
- <para>
- The <productname>PostgreSQL</productname> query planner relies on
- statistical information about the contents of tables in order to
- generate good plans for queries. These statistics are gathered by
- the <link linkend="sql-analyze"><command>ANALYZE</command></link> command,
- which can be invoked by itself or
- as an optional step in <command>VACUUM</command>. It is important to have
- reasonably accurate statistics, otherwise poor choices of plans might
- degrade database performance.
- </para>
-
- <para>
- The autovacuum daemon, if enabled, will automatically issue
- <command>ANALYZE</command> commands whenever the content of a table has
- changed sufficiently. However, administrators might prefer to rely
- on manually-scheduled <command>ANALYZE</command> operations, particularly
- if it is known that update activity on a table will not affect the
- statistics of <quote>interesting</quote> columns. The daemon schedules
- <command>ANALYZE</command> strictly as a function of the number of rows
- inserted or updated; it has no knowledge of whether that will lead
- to meaningful statistical changes.
- </para>
-
- <para>
- Tuples changed in partitions and inheritance children do not trigger
- analyze on the parent table. If the parent table is empty or rarely
- changed, it may never be processed by autovacuum, and the statistics for
- the inheritance tree as a whole won't be collected. It is necessary to
- run <command>ANALYZE</command> on the parent table manually in order to
- keep the statistics up to date.
- </para>
-
- <para>
- As with vacuuming for space recovery, frequent updates of statistics
- are more useful for heavily-updated tables than for seldom-updated
- ones. But even for a heavily-updated table, there might be no need for
- statistics updates if the statistical distribution of the data is
- not changing much. A simple rule of thumb is to think about how much
- the minimum and maximum values of the columns in the table change.
- For example, a <type>timestamp</type> column that contains the time
- of row update will have a constantly-increasing maximum value as
- rows are added and updated; such a column will probably need more
- frequent statistics updates than, say, a column containing URLs for
- pages accessed on a website. The URL column might receive changes just
- as often, but the statistical distribution of its values probably
- changes relatively slowly.
- </para>
-
- <para>
- It is possible to run <command>ANALYZE</command> on specific tables and even
- just specific columns of a table, so the flexibility exists to update some
- statistics more frequently than others if your application requires it.
- In practice, however, it is usually best to just analyze the entire
- database, because it is a fast operation. <command>ANALYZE</command> uses a
- statistically random sampling of the rows of a table rather than reading
- every single row.
- </para>
-
- <tip>
- <para>
- Although per-column tweaking of <command>ANALYZE</command> frequency might not be
- very productive, you might find it worthwhile to do per-column
- adjustment of the level of detail of the statistics collected by
- <command>ANALYZE</command>. Columns that are heavily used in <literal>WHERE</literal>
- clauses and have highly irregular data distributions might require a
- finer-grain data histogram than other columns. See <command>ALTER TABLE
- SET STATISTICS</command>, or change the database-wide default using the <xref
- linkend="guc-default-statistics-target"/> configuration parameter.
- </para>
-
- <para>
- Also, by default there is limited information available about
- the selectivity of functions. However, if you create a statistics
- object or an expression
- index that uses a function call, useful statistics will be
- gathered about the function, which can greatly improve query
- plans that use the expression index.
- </para>
- </tip>
-
- <tip>
- <para>
- The autovacuum daemon does not issue <command>ANALYZE</command> commands for
- foreign tables, since it has no means of determining how often that
- might be useful. If your queries require statistics on foreign tables
- for proper planning, it's a good idea to run manually-managed
- <command>ANALYZE</command> commands on those tables on a suitable schedule.
- </para>
- </tip>
-
- <tip>
- <para>
- The autovacuum daemon does not issue <command>ANALYZE</command> commands
- for partitioned tables. Inheritance parents will only be analyzed if the
- parent itself is changed - changes to child tables do not trigger
- autoanalyze on the parent table. If your queries require statistics on
- parent tables for proper planning, it is necessary to periodically run
- a manual <command>ANALYZE</command> on those tables to keep the statistics
- up to date.
- </para>
- </tip>
-
- </sect2>
-
- <sect2 id="vacuum-for-visibility-map">
- <title>Updating the Visibility Map</title>
-
- <para>
- Vacuum maintains a <link linkend="storage-vm">visibility map</link> for each
- table to keep track of which pages contain only tuples that are known to be
- visible to all active transactions (and all future transactions, until the
- page is again modified). This has two purposes. First, vacuum
- itself can skip such pages on the next run, since there is nothing to
- clean up.
- </para>
-
- <para>
- Second, it allows <productname>PostgreSQL</productname> to answer some
- queries using only the index, without reference to the underlying table.
- Since <productname>PostgreSQL</productname> indexes don't contain tuple
- visibility information, a normal index scan fetches the heap tuple for each
- matching index entry, to check whether it should be seen by the current
- transaction.
- An <link linkend="indexes-index-only-scans"><firstterm>index-only
- scan</firstterm></link>, on the other hand, checks the visibility map first.
- If it's known that all tuples on the page are
- visible, the heap fetch can be skipped. This is most useful on
- large data sets where the visibility map can prevent disk accesses.
- The visibility map is vastly smaller than the heap, so it can easily be
- cached even when the heap is very large.
- </para>
- </sect2>
-
<sect2 id="vacuum-for-wraparound">
<title>Preventing Transaction ID Wraparound Failures</title>
@@ -933,7 +788,152 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect3>
</sect2>
- </sect1>
+
+ <sect2 id="vacuum-for-visibility-map">
+ <title>Updating the Visibility Map</title>
+
+ <para>
+ Vacuum maintains a <link linkend="storage-vm">visibility
+ map</link> for each table to keep track of which pages contain
+ only tuples that are known to be visible to all active
+ transactions (and all future transactions, until the page is again
+ modified). This has two purposes. First, vacuum itself can skip
+ such pages on the next run, since there is nothing to clean up.
+ </para>
+
+ <para>
+ Second, it allows <productname>PostgreSQL</productname> to answer
+ some queries using only the index, without reference to the
+ underlying table. Since <productname>PostgreSQL</productname>
+ indexes don't contain tuple visibility information, a normal index
+ scan fetches the heap tuple for each matching index entry, to
+ check whether it should be seen by the current transaction. An
+ <link linkend="indexes-index-only-scans"><firstterm>index-only
+ scan</firstterm></link>, on the other hand, checks the
+ visibility map first. If it's known that all tuples on the page
+ are visible, the heap fetch can be skipped. This is most useful
+ on large data sets where the visibility map can prevent disk
+ accesses. The visibility map is vastly smaller than the heap, so
+ it can easily be cached even when the heap is very large.
+ </para>
+ </sect2>
+
+ <sect2 id="vacuum-for-statistics">
+ <title>Updating Planner Statistics</title>
+
+ <indexterm zone="vacuum-for-statistics">
+ <primary>statistics</primary>
+ <secondary>of the planner</secondary>
+ </indexterm>
+
+ <indexterm zone="vacuum-for-statistics">
+ <primary>ANALYZE</primary>
+ </indexterm>
+
+ <para>
+ The <productname>PostgreSQL</productname> query planner relies on
+ statistical information about the contents of tables in order to
+ generate good plans for queries. These statistics are gathered by
+ the <link linkend="sql-analyze"><command>ANALYZE</command></link> command,
+ which can be invoked by itself or
+ as an optional step in <command>VACUUM</command>. It is important to have
+ reasonably accurate statistics, otherwise poor choices of plans might
+ degrade database performance.
+ </para>
+
+ <para>
+ The autovacuum daemon, if enabled, will automatically issue
+ <command>ANALYZE</command> commands whenever the content of a table has
+ changed sufficiently. However, administrators might prefer to rely
+ on manually-scheduled <command>ANALYZE</command> operations, particularly
+ if it is known that update activity on a table will not affect the
+ statistics of <quote>interesting</quote> columns. The daemon schedules
+ <command>ANALYZE</command> strictly as a function of the number of rows
+ inserted or updated; it has no knowledge of whether that will lead
+ to meaningful statistical changes.
+ </para>
+
+ <para>
+ Tuples changed in partitions and inheritance children do not trigger
+ analyze on the parent table. If the parent table is empty or rarely
+ changed, it may never be processed by autovacuum, and the statistics for
+ the inheritance tree as a whole won't be collected. It is necessary to
+ run <command>ANALYZE</command> on the parent table manually in order to
+ keep the statistics up to date.
+ </para>
+
+ <para>
+ As with vacuuming for space recovery, frequent updates of statistics
+ are more useful for heavily-updated tables than for seldom-updated
+ ones. But even for a heavily-updated table, there might be no need for
+ statistics updates if the statistical distribution of the data is
+ not changing much. A simple rule of thumb is to think about how much
+ the minimum and maximum values of the columns in the table change.
+ For example, a <type>timestamp</type> column that contains the time
+ of row update will have a constantly-increasing maximum value as
+ rows are added and updated; such a column will probably need more
+ frequent statistics updates than, say, a column containing URLs for
+ pages accessed on a website. The URL column might receive changes just
+ as often, but the statistical distribution of its values probably
+ changes relatively slowly.
+ </para>
+
+ <para>
+ It is possible to run <command>ANALYZE</command> on specific tables and even
+ just specific columns of a table, so the flexibility exists to update some
+ statistics more frequently than others if your application requires it.
+ In practice, however, it is usually best to just analyze the entire
+ database, because it is a fast operation. <command>ANALYZE</command> uses a
+ statistically random sampling of the rows of a table rather than reading
+ every single row.
+ </para>
+
+ <tip>
+ <para>
+ Although per-column tweaking of <command>ANALYZE</command> frequency might not be
+ very productive, you might find it worthwhile to do per-column
+ adjustment of the level of detail of the statistics collected by
+ <command>ANALYZE</command>. Columns that are heavily used in <literal>WHERE</literal>
+ clauses and have highly irregular data distributions might require a
+ finer-grain data histogram than other columns. See <command>ALTER TABLE
+ SET STATISTICS</command>, or change the database-wide default using the <xref
+ linkend="guc-default-statistics-target"/> configuration parameter.
+ </para>
+
+ <para>
+ Also, by default there is limited information available about
+ the selectivity of functions. However, if you create a statistics
+ object or an expression
+ index that uses a function call, useful statistics will be
+ gathered about the function, which can greatly improve query
+ plans that use the expression index.
+ </para>
+ </tip>
+
+ <tip>
+ <para>
+ The autovacuum daemon does not issue <command>ANALYZE</command> commands for
+ foreign tables, since it has no means of determining how often that
+ might be useful. If your queries require statistics on foreign tables
+ for proper planning, it's a good idea to run manually-managed
+ <command>ANALYZE</command> commands on those tables on a suitable schedule.
+ </para>
+ </tip>
+
+ <tip>
+ <para>
+ The autovacuum daemon does not issue <command>ANALYZE</command> commands
+ for partitioned tables. Inheritance parents will only be analyzed if the
+ parent itself is changed - changes to child tables do not trigger
+ autoanalyze on the parent table. If your queries require statistics on
+ parent tables for proper planning, it is necessary to periodically run
+ a manual <command>ANALYZE</command> on those tables to keep the statistics
+ up to date.
+ </para>
+ </tip>
+
+ </sect2>
+</sect1>
<sect1 id="routine-reindex">
--
2.40.1
[application/octet-stream] v3-0003-Normalize-maintenance.sgml-indentation.patch (4.7K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/10-v3-0003-Normalize-maintenance.sgml-indentation.patch)
download | inline diff:
From 93fc0893e4ca17068737d14e4a9901c83b264e99 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 15:20:13 -0700
Subject: [PATCH v3 3/9] Normalize maintenance.sgml indentation.
---
doc/src/sgml/maintenance.sgml | 82 +++++++++++++++++------------------
1 file changed, 41 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 6a7ec7c1d..e8c8647cd 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -11,53 +11,53 @@
<primary>routine maintenance</primary>
</indexterm>
- <para>
- <productname>PostgreSQL</productname>, like any database software, requires that certain tasks
- be performed regularly to achieve optimum performance. The tasks
- discussed here are <emphasis>required</emphasis>, but they
- are repetitive in nature and can easily be automated using standard
- tools such as <application>cron</application> scripts or
- Windows' <application>Task Scheduler</application>. It is the database
- administrator's responsibility to set up appropriate scripts, and to
- check that they execute successfully.
- </para>
+ <para>
+ <productname>PostgreSQL</productname>, like any database software, requires that certain tasks
+ be performed regularly to achieve optimum performance. The tasks
+ discussed here are <emphasis>required</emphasis>, but they
+ are repetitive in nature and can easily be automated using standard
+ tools such as <application>cron</application> scripts or
+ Windows' <application>Task Scheduler</application>. It is the database
+ administrator's responsibility to set up appropriate scripts, and to
+ check that they execute successfully.
+ </para>
- <para>
- One obvious maintenance task is the creation of backup copies of the data on a
- regular schedule. Without a recent backup, you have no chance of recovery
- after a catastrophe (disk failure, fire, mistakenly dropping a critical
- table, etc.). The backup and recovery mechanisms available in
- <productname>PostgreSQL</productname> are discussed at length in
- <xref linkend="backup"/>.
- </para>
+ <para>
+ One obvious maintenance task is the creation of backup copies of the data on a
+ regular schedule. Without a recent backup, you have no chance of recovery
+ after a catastrophe (disk failure, fire, mistakenly dropping a critical
+ table, etc.). The backup and recovery mechanisms available in
+ <productname>PostgreSQL</productname> are discussed at length in
+ <xref linkend="backup"/>.
+ </para>
- <para>
- The other main category of maintenance task is periodic <quote>vacuuming</quote>
- of the database. This activity is discussed in
- <xref linkend="routine-vacuuming"/>. Closely related to this is updating
- the statistics that will be used by the query planner, as discussed in
- <xref linkend="vacuum-for-statistics"/>.
- </para>
+ <para>
+ The other main category of maintenance task is periodic <quote>vacuuming</quote>
+ of the database. This activity is discussed in
+ <xref linkend="routine-vacuuming"/>. Closely related to this is updating
+ the statistics that will be used by the query planner, as discussed in
+ <xref linkend="vacuum-for-statistics"/>.
+ </para>
- <para>
- Another task that might need periodic attention is log file management.
- This is discussed in <xref linkend="logfile-maintenance"/>.
- </para>
+ <para>
+ Another task that might need periodic attention is log file management.
+ This is discussed in <xref linkend="logfile-maintenance"/>.
+ </para>
- <para>
- <ulink
+ <para>
+ <ulink
url="https://bucardo.org/check_postgres/"><application>check_postgres</application></ulink>
- is available for monitoring database health and reporting unusual
- conditions. <application>check_postgres</application> integrates with
- Nagios and MRTG, but can be run standalone too.
- </para>
+ is available for monitoring database health and reporting unusual
+ conditions. <application>check_postgres</application> integrates with
+ Nagios and MRTG, but can be run standalone too.
+ </para>
- <para>
- <productname>PostgreSQL</productname> is low-maintenance compared
- to some other database management systems. Nonetheless,
- appropriate attention to these tasks will go far towards ensuring a
- pleasant and productive experience with the system.
- </para>
+ <para>
+ <productname>PostgreSQL</productname> is low-maintenance compared
+ to some other database management systems. Nonetheless,
+ appropriate attention to these tasks will go far towards ensuring a
+ pleasant and productive experience with the system.
+ </para>
<sect1 id="autovacuum">
<title>The Autovacuum Daemon</title>
--
2.40.1
[application/octet-stream] v3-0002-Restructure-autovacuum-daemon-section.patch (5.3K, ../../CAH2-WznyY5hUgdYJE8c5KoS2QbGXH9ggEuux-xY-gx5ETa4PAg@mail.gmail.com/11-v3-0002-Restructure-autovacuum-daemon-section.patch)
download | inline diff:
From b73a948bf0723ce424cf75ba23694e78c4c3aefd Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Mon, 24 Apr 2023 09:21:01 -0700
Subject: [PATCH v3 2/9] Restructure autovacuum daemon section.
Add sect2/sect3 subsections to autovacuum sect1. Also reorder the
content slightly for clarity.
TODO Add some basic explanations of vacuuming and relfrozenxid
advancement, since that now appears later on in the chapter.
Alternatively, move the autovacuum daemon sect1 after the routine
vacuuming sect1.
---
doc/src/sgml/maintenance.sgml | 66 ++++++++++++++++++++++-------------
1 file changed, 42 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index a6295c399..6a7ec7c1d 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -100,6 +100,8 @@
autovacuum workers' activity.
</para>
+ <sect2 id="autovacuum-scheduling">
+ <title>Autovacuum Scheduling</title>
<para>
If several large tables all become eligible for vacuuming in a short
amount of time, all autovacuum workers might become occupied with
@@ -112,6 +114,8 @@
<xref linkend="guc-superuser-reserved-connections"/> limits.
</para>
+ <sect3 id="autovacuum-vacuum-thresholds">
+ <title>Configurable thresholds for vacuuming</title>
<para>
Tables whose <structfield>relfrozenxid</structfield> value is more than
<xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
@@ -159,7 +163,10 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac
<structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
since the last vacuum are scanned.
</para>
+ </sect3>
+ <sect3 id="autovacuum-analyze-thresholds">
+ <title>Configurable thresholds for <command>ANALYZE</command></title>
<para>
For analyze, a similar condition is used: the threshold, defined as:
<programlisting>
@@ -168,20 +175,6 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
is compared to the total number of tuples inserted, updated, or deleted
since the last <command>ANALYZE</command>.
</para>
-
- <para>
- Partitioned tables are not processed by autovacuum. Statistics
- should be collected by running a manual <command>ANALYZE</command> when it is
- first populated, and again whenever the distribution of data in its
- partitions changes significantly.
- </para>
-
- <para>
- Temporary tables cannot be accessed by autovacuum. Therefore,
- appropriate vacuum and analyze operations should be performed via
- session SQL commands.
- </para>
-
<para>
The default thresholds and scale factors are taken from
<filename>postgresql.conf</filename>, but it is possible to override them
@@ -192,18 +185,25 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
used. See <xref linkend="runtime-config-autovacuum"/> for more details on
the global settings.
</para>
+ </sect3>
+ </sect2>
- <para>
- When multiple workers are running, the autovacuum cost delay parameters
- (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
- <quote>balanced</quote> among all the running workers, so that the
- total I/O impact on the system is the same regardless of the number
- of workers actually running. However, any workers processing tables whose
- per-table <literal>autovacuum_vacuum_cost_delay</literal> or
- <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
- are not considered in the balancing algorithm.
- </para>
+ <sect2 id="autovacuum-cost-delays">
+ <title>Autovacuum Cost-based Delays</title>
+ <para>
+ When multiple workers are running, the autovacuum cost delay parameters
+ (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
+ <quote>balanced</quote> among all the running workers, so that the
+ total I/O impact on the system is the same regardless of the number
+ of workers actually running. However, any workers processing tables whose
+ per-table <literal>autovacuum_vacuum_cost_delay</literal> or
+ <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
+ are not considered in the balancing algorithm.
+ </para>
+ </sect2>
+ <sect2 id="autovacuum-lock-conflicts">
+ <title>Autovacuum and Lock Conflicts</title>
<para>
Autovacuum workers generally don't block other commands. If a process
attempts to acquire a lock that conflicts with the
@@ -223,6 +223,24 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
effectively prevent autovacuums from ever completing.
</para>
</warning>
+ </sect2>
+
+ <sect2 id="autovacuum-limitations">
+ <title>Limitations</title>
+ <para>
+ Partitioned tables are not processed by autovacuum. Statistics
+ should be collected by running a manual <command>ANALYZE</command> when it is
+ first populated, and again whenever the distribution of data in its
+ partitions changes significantly.
+ </para>
+
+ <para>
+ Temporary tables cannot be accessed by autovacuum. Therefore,
+ appropriate vacuum and analyze operations should be performed via
+ session SQL commands.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="routine-vacuuming">
--
2.40.1
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
@ 2023-05-12 01:18 ` Peter Geoghegan <[email protected]>
2023-05-12 17:36 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-12 01:18 UTC (permalink / raw)
To: samay sharma <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Thu, May 4, 2023 at 3:18 PM samay sharma <[email protected]> wrote:
> What do you think about the term "Exhaustion"? Maybe something like "XID allocation exhaustion" or "Exhaustion of allocatable XIDs"?
I use the term "transaction ID exhaustion" in the attached revision,
v4. Overall, v4 builds on the work that went into v2 and v3, by
continuing to polish the overhaul of everything related to freezing,
relfrozenxid advancement, and anti-wraparound autovacuum.
It would be nice if it was possible to add an animation/diagram a
little like this one: https://tuple-freezing-demo.angusd.com (this is
how I tend to think about the "transaction ID space".)
I feel that the patch that deals with freezing is really coming
together in v4. The main problem now is lack of detailed review --
though the freezing related patch is still not committable, it's
getting close now. (The changes to the docs covering freezing should
be committed separately from any further work on "25.2.1. Recovering
Disk Space". I still haven't done much there in v4, and those parts
clearly aren't anywhere near being committable. So, for now, they can
mostly be ignored.)
v4 also limits use of the term "wraparound" to places that directly
discuss anti-wraparound autovacuums (plus one place in xact.sgml,
where discussion of "true unsigned integer wraparound" and related
implementation details has been moved). Otherwise we use the term
"transaction ID exhaustion", which is pretty much the user-facing name
for "xidStopLimit". I feel that this is a huge improvement, for the
reason given to Greg earlier. I'm flexible on the details, but I feel
strongly that we should minimize use of the term wraparound wherever
it might have the connotation of "the past becoming the future". This
is not a case of inventing a new terminology for its own sake. If
anybody is skeptical I ask that they take a look at what I came up
with before declaring it a bad idea. I have made that as easy as
possible, by once again attaching a prebuilt routine-vacuuming.html.
I no longer believe that committing this patch series needs to block
on the patch that seeks to put things straight with single user mode
and xidStopLimit/transaction ID exhaustion (the one that John Naylor
is currently working on getting in shape), either (I'll explain my
reasoning if somebody wants to hear it).
Other changes in v4, compared to v3:
* Improved discussion of the differences between non-aggressive and
aggressive VACUUM.
Now mentions the issue of aggressive VACUUMs waiting for a cleanup
lock, including mention of the BufferPin wait event. This is the
second, minor difference between each kind of VACUUM. It matters much
less than the first difference, but it does merit a mention.
The discussion of aggressive VACUUM seems to be best approached by
starting with the mechanical differences, and only later going into
the consequences of those differences. (Particularly catch-up
freezing.)
* Explains "catch-up freezing" performed by aggressive VACUUMs directly.
"Catch-up" freezing is the really important "consequence" -- something
that emerges from how each type of VACUUM behaves over time. It is an
indirect consequence of the behaviors. I would like to counter the
perception that some users have about freezing only happening during
aggressive VACUUMs (or anti-wraparound autovacuums). But more than
that, talking about catch-up freezing seems essential because it is
the single most important difference.
* Much improved handling of the discussion of anti-wraparound
autovacuum, and how it relates to aggressive VACUUMs, following
feedback from Samay.
There is now only fairly minimal overlap in the discussion of
aggressive VACUUM and anti-wraparound autovacuuming. We finish the
discussion of aggressive VACUUM just after we start discussing
anti-wraparound autovacuum. This transition works well, because it
enforces the idea that anti-wraparound autovacuum isn't really special
compared to any other aggressive autovacuum. This was something that
Samay expressed particularly concern about: making anti-wraparound
autovacuums sound less scary. Though it's also a concern I had from
the outset, based on practical experience and interactions with people
that have much less knowledge of Postgres than I do.
* Anti-wraparound autovacuum is now mostly discussed as something that
happens to static or mostly-static tables.
This is related to the goal of making anti-wraparound autovacuums
sound less scary. Larger tables don't necessarily require any
anti-wraparound autovacuums these days -- we have the insert-driven
autovacuum trigger condition these days, so it's plausible (perhaps
even likely) that all aggressive VACUUMs against the largest
append-only tables can happen when autovacuum triggers VACUUMs to
processed recently inserted tuples.
This moves discussion of anti-wraparound av in the direction of:
"Anti-wraparound autovacuum is a special type of autovacuum. Its
purpose is to ensure that relfrozenxid advances when no earlier VACUUM
could advance it in passing — often because no VACUUM has run against
the table for an extended period."
* Added a couple of "Tips" about instrumentation that appears in the
server log whenever autovacuum reports on a VACUUM operation.
* Much improved "Truncating Transaction Status Information" subsection.
My explanation of the ways in which autovacuum_freeze_max_age can
affect the storage overhead of commit/abort status in pg_xact is much
clearer than it was in v3 -- pg_xact truncation is now treated as
something loosely related to the global config of anti-wraparound
autovacuum, which makes most sense.
It took a great deal of effort to find a structure that covered
everything, and that highlighted all of the important relationships
without going too far, while at the same time not being a huge mess.
That's what I feel I've arrived at with v4.
--
Peter Geoghegan
Attachments:
[text/html] routine-vacuuming.html (73.1K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/2-routine-vacuuming.html)
download
[application/octet-stream] v4-0003-Reindent-autovacuum-daemon-sect1.patch (13.4K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/3-v4-0003-Reindent-autovacuum-daemon-sect1.patch)
download | inline diff:
From 3d246ace93f13594b8525b3516f35560f9721599 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 6 May 2023 12:32:43 -0700
Subject: [PATCH v4 3/9] Reindent autovacuum daemon sect1.
When "The Autovacuum Daemon" became its own sect1, the sect1 tags were
left indented incorrectly (for a sect1) in order to make the initial
"move text into its own sect1" commit as mechanical as possible (we kept
the original sect2 indentation). A later commit further split up "The
Autovacuum Daemon" into further sect2 and sect3 subsections, to add
useful headings. Now we actually fix the indentation changes that were
put off at first.
It may look like these indentation changes are wrong (they look wrong
relative to the chapter-level introductory content that precedes these
changes to indentation), but it's actually the other way around: there
is a preexisting problem with the indentation for the chapter-level
introductory material (which I have opted to not fix now).
Note that the newly added "The Autovacuum Daemon" sect1 now has one
space of indentation at the top level, just like every other sect1.
As usual, the goal of structuring things this way is to make life easier
for translators.
---
doc/src/sgml/maintenance.sgml | 198 +++++++++++++++++-----------------
1 file changed, 99 insertions(+), 99 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index c27efc58d..702e2797c 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -59,46 +59,46 @@
pleasant and productive experience with the system.
</para>
- <sect1 id="autovacuum">
- <title>The Autovacuum Daemon</title>
+ <sect1 id="autovacuum">
+ <title>The Autovacuum Daemon</title>
- <indexterm>
- <primary>autovacuum</primary>
- <secondary>general information</secondary>
- </indexterm>
- <para>
- <productname>PostgreSQL</productname> has an optional but highly
- recommended feature called <firstterm>autovacuum</firstterm>,
- whose purpose is to automate the execution of
- <command>VACUUM</command> and <command>ANALYZE</command> commands.
- When enabled, autovacuum checks for
- tables that have had a large number of inserted, updated or deleted
- tuples. These checks use the statistics collection facility;
- therefore, autovacuum cannot be used unless <xref
+ <indexterm>
+ <primary>autovacuum</primary>
+ <secondary>general information</secondary>
+ </indexterm>
+ <para>
+ <productname>PostgreSQL</productname> has an optional but highly
+ recommended feature called <firstterm>autovacuum</firstterm>,
+ whose purpose is to automate the execution of
+ <command>VACUUM</command> and <command>ANALYZE</command> commands.
+ When enabled, autovacuum checks for
+ tables that have had a large number of inserted, updated or deleted
+ tuples. These checks use the statistics collection facility;
+ therefore, autovacuum cannot be used unless <xref
linkend="guc-track-counts"/> is set to <literal>true</literal>.
- In the default configuration, autovacuuming is enabled and the related
- configuration parameters are appropriately set.
- </para>
+ In the default configuration, autovacuuming is enabled and the related
+ configuration parameters are appropriately set.
+ </para>
- <para>
- The <quote>autovacuum daemon</quote> actually consists of multiple processes.
- There is a persistent daemon process, called the
- <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
- <firstterm>autovacuum worker</firstterm> processes for all databases. The
- launcher will distribute the work across time, attempting to start one
- worker within each database every <xref linkend="guc-autovacuum-naptime"/>
- seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
- a new worker will be launched every
- <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
- A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
- are allowed to run at the same time. If there are more than
- <varname>autovacuum_max_workers</varname> databases to be processed,
- the next database will be processed as soon as the first worker finishes.
- Each worker process will check each table within its database and
- execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
- <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
- autovacuum workers' activity.
- </para>
+ <para>
+ The <quote>autovacuum daemon</quote> actually consists of multiple processes.
+ There is a persistent daemon process, called the
+ <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
+ <firstterm>autovacuum worker</firstterm> processes for all databases. The
+ launcher will distribute the work across time, attempting to start one
+ worker within each database every <xref linkend="guc-autovacuum-naptime"/>
+ seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
+ a new worker will be launched every
+ <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
+ A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
+ are allowed to run at the same time. If there are more than
+ <varname>autovacuum_max_workers</varname> databases to be processed,
+ the next database will be processed as soon as the first worker finishes.
+ Each worker process will check each table within its database and
+ execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
+ <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
+ autovacuum workers' activity.
+ </para>
<sect2 id="autovacuum-scheduling">
<title>Autovacuum Scheduling</title>
@@ -114,78 +114,78 @@
<xref linkend="guc-superuser-reserved-connections"/> limits.
</para>
- <sect3 id="autovacuum-vacuum-thresholds">
- <title>Configurable thresholds for vacuuming</title>
- <para>
- Tables whose <structfield>relfrozenxid</structfield> value is more than
- <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
- vacuumed (this also applies to those tables whose freeze max age has
- been modified via storage parameters; see below). Otherwise, if the
- number of tuples obsoleted since the last
- <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
- table is vacuumed. The vacuum threshold is defined as:
+ <sect3 id="autovacuum-vacuum-thresholds">
+ <title>Configurable thresholds for vacuuming</title>
+ <para>
+ Tables whose <structfield>relfrozenxid</structfield> value is more than
+ <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
+ vacuumed (this also applies to those tables whose freeze max age has
+ been modified via storage parameters; see below). Otherwise, if the
+ number of tuples obsoleted since the last
+ <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
+ table is vacuumed. The vacuum threshold is defined as:
<programlisting>
vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
</programlisting>
- where the vacuum base threshold is
- <xref linkend="guc-autovacuum-vacuum-threshold"/>,
- the vacuum scale factor is
- <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
- and the number of tuples is
- <structname>pg_class</structname>.<structfield>reltuples</structfield>.
- </para>
+ where the vacuum base threshold is
+ <xref linkend="guc-autovacuum-vacuum-threshold"/>,
+ the vacuum scale factor is
+ <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
+ and the number of tuples is
+ <structname>pg_class</structname>.<structfield>reltuples</structfield>.
+ </para>
- <para>
- The table is also vacuumed if the number of tuples inserted since the last
- vacuum has exceeded the defined insert threshold, which is defined as:
+ <para>
+ The table is also vacuumed if the number of tuples inserted since the last
+ vacuum has exceeded the defined insert threshold, which is defined as:
<programlisting>
vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
</programlisting>
- where the vacuum insert base threshold is
- <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
- and vacuum insert scale factor is
- <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
- Such vacuums may allow portions of the table to be marked as
- <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
- can reduce the work required in subsequent vacuums.
- For tables which receive <command>INSERT</command> operations but no or
- almost no <command>UPDATE</command>/<command>DELETE</command> operations,
- it may be beneficial to lower the table's
- <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
- tuples to be frozen by earlier vacuums. The number of obsolete tuples and
- the number of inserted tuples are obtained from the cumulative statistics system;
- it is a semi-accurate count updated by each <command>UPDATE</command>,
- <command>DELETE</command> and <command>INSERT</command> operation. (It is
- only semi-accurate because some information might be lost under heavy
- load.) If the <structfield>relfrozenxid</structfield> value of the table
- is more than <varname>vacuum_freeze_table_age</varname> transactions old,
- an aggressive vacuum is performed to freeze old tuples and advance
- <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
- since the last vacuum are scanned.
- </para>
- </sect3>
+ where the vacuum insert base threshold is
+ <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
+ and vacuum insert scale factor is
+ <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
+ Such vacuums may allow portions of the table to be marked as
+ <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
+ can reduce the work required in subsequent vacuums.
+ For tables which receive <command>INSERT</command> operations but no or
+ almost no <command>UPDATE</command>/<command>DELETE</command> operations,
+ it may be beneficial to lower the table's
+ <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
+ tuples to be frozen by earlier vacuums. The number of obsolete tuples and
+ the number of inserted tuples are obtained from the cumulative statistics system;
+ it is a semi-accurate count updated by each <command>UPDATE</command>,
+ <command>DELETE</command> and <command>INSERT</command> operation. (It is
+ only semi-accurate because some information might be lost under heavy
+ load.) If the <structfield>relfrozenxid</structfield> value of the table
+ is more than <varname>vacuum_freeze_table_age</varname> transactions old,
+ an aggressive vacuum is performed to freeze old tuples and advance
+ <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
+ since the last vacuum are scanned.
+ </para>
+ </sect3>
- <sect3 id="autovacuum-analyze-thresholds">
- <title>Configurable thresholds for <command>ANALYZE</command></title>
- <para>
- For analyze, a similar condition is used: the threshold, defined as:
+ <sect3 id="autovacuum-analyze-thresholds">
+ <title>Configurable thresholds for <command>ANALYZE</command></title>
+ <para>
+ For analyze, a similar condition is used: the threshold, defined as:
<programlisting>
analyze threshold = analyze base threshold + analyze scale factor * number of tuples
</programlisting>
- is compared to the total number of tuples inserted, updated, or deleted
- since the last <command>ANALYZE</command>.
- </para>
- <para>
- The default thresholds and scale factors are taken from
- <filename>postgresql.conf</filename>, but it is possible to override them
- (and many other autovacuum control parameters) on a per-table basis; see
- <xref linkend="sql-createtable-storage-parameters"/> for more information.
- If a setting has been changed via a table's storage parameters, that value
- is used when processing that table; otherwise the global settings are
- used. See <xref linkend="runtime-config-autovacuum"/> for more details on
- the global settings.
- </para>
- </sect3>
+ is compared to the total number of tuples inserted, updated, or deleted
+ since the last <command>ANALYZE</command>.
+ </para>
+ <para>
+ The default thresholds and scale factors are taken from
+ <filename>postgresql.conf</filename>, but it is possible to override them
+ (and many other autovacuum control parameters) on a per-table basis; see
+ <xref linkend="sql-createtable-storage-parameters"/> for more information.
+ If a setting has been changed via a table's storage parameters, that value
+ is used when processing that table; otherwise the global settings are
+ used. See <xref linkend="runtime-config-autovacuum"/> for more details on
+ the global settings.
+ </para>
+ </sect3>
</sect2>
<sect2 id="autovacuum-cost-delays">
@@ -240,7 +240,7 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
session SQL commands.
</para>
</sect2>
- </sect1>
+ </sect1>
<sect1 id="routine-vacuuming">
<title>Routine Vacuuming</title>
--
2.40.1
[application/octet-stream] v4-0002-Restructure-autovacuum-daemon-section.patch (4.7K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/4-v4-0002-Restructure-autovacuum-daemon-section.patch)
download | inline diff:
From b41c88098364b10420bc5499cc8775f45406742c Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Mon, 24 Apr 2023 09:21:01 -0700
Subject: [PATCH v4 2/9] Restructure autovacuum daemon section.
Add sect2/sect3 subsections to autovacuum sect1. Also reorder the
content slightly for clarity by consolidating "limitations".
The next commit finishes recent changes to the autovacuum daemon content
by reindenting everything.
TODO Add some basic explanations of vacuuming and relfrozenxid
advancement, since that now appears later on in the chapter.
Alternatively, move the autovacuum daemon sect1 after the routine
vacuuming sect1.
---
doc/src/sgml/maintenance.sgml | 45 ++++++++++++++++++++++++-----------
1 file changed, 31 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index b9091e72c..c27efc58d 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -100,6 +100,8 @@
autovacuum workers' activity.
</para>
+ <sect2 id="autovacuum-scheduling">
+ <title>Autovacuum Scheduling</title>
<para>
If several large tables all become eligible for vacuuming in a short
amount of time, all autovacuum workers might become occupied with
@@ -112,6 +114,8 @@
<xref linkend="guc-superuser-reserved-connections"/> limits.
</para>
+ <sect3 id="autovacuum-vacuum-thresholds">
+ <title>Configurable thresholds for vacuuming</title>
<para>
Tables whose <structfield>relfrozenxid</structfield> value is more than
<xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
@@ -159,7 +163,10 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac
<structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
since the last vacuum are scanned.
</para>
+ </sect3>
+ <sect3 id="autovacuum-analyze-thresholds">
+ <title>Configurable thresholds for <command>ANALYZE</command></title>
<para>
For analyze, a similar condition is used: the threshold, defined as:
<programlisting>
@@ -168,20 +175,6 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
is compared to the total number of tuples inserted, updated, or deleted
since the last <command>ANALYZE</command>.
</para>
-
- <para>
- Partitioned tables are not processed by autovacuum. Statistics
- should be collected by running a manual <command>ANALYZE</command> when it is
- first populated, and again whenever the distribution of data in its
- partitions changes significantly.
- </para>
-
- <para>
- Temporary tables cannot be accessed by autovacuum. Therefore,
- appropriate vacuum and analyze operations should be performed via
- session SQL commands.
- </para>
-
<para>
The default thresholds and scale factors are taken from
<filename>postgresql.conf</filename>, but it is possible to override them
@@ -192,7 +185,11 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
used. See <xref linkend="runtime-config-autovacuum"/> for more details on
the global settings.
</para>
+ </sect3>
+ </sect2>
+ <sect2 id="autovacuum-cost-delays">
+ <title>Autovacuum Cost-based Delays</title>
<para>
When multiple workers are running, the autovacuum cost delay parameters
(see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
@@ -203,7 +200,10 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
are not considered in the balancing algorithm.
</para>
+ </sect2>
+ <sect2 id="autovacuum-lock-conflicts">
+ <title>Autovacuum and Lock Conflicts</title>
<para>
Autovacuum workers generally don't block other commands. If a process
attempts to acquire a lock that conflicts with the
@@ -223,6 +223,23 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
effectively prevent autovacuums from ever completing.
</para>
</warning>
+ </sect2>
+
+ <sect2 id="autovacuum-limitations">
+ <title>Limitations</title>
+ <para>
+ Partitioned tables are not processed by autovacuum. Statistics
+ should be collected by running a manual <command>ANALYZE</command> when it is
+ first populated, and again whenever the distribution of data in its
+ partitions changes significantly.
+ </para>
+
+ <para>
+ Temporary tables cannot be accessed by autovacuum. Therefore,
+ appropriate vacuum and analyze operations should be performed via
+ session SQL commands.
+ </para>
+ </sect2>
</sect1>
<sect1 id="routine-vacuuming">
--
2.40.1
[application/octet-stream] v4-0001-Make-autovacuum-docs-into-a-sect1-of-its-own.patch (18.9K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/5-v4-0001-Make-autovacuum-docs-into-a-sect1-of-its-own.patch)
download | inline diff:
From cd2b42a6e986fcf70269cb14050c5113f6b84ee0 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Wed, 12 Apr 2023 14:42:06 -0700
Subject: [PATCH v4 1/9] Make autovacuum docs into a sect1 of its own.
This doesn't change any of the content itself. Though it does move it
from the end of "Routine Vacuuming" (which is itself a sect1) to a whole
new sect1 that appears _before_ "Routine Vacuuming".
This commit is as mechanical as possible, in the sense that it is
structured in such a way as to make git diff's "dimmed-zebra" mode show
as few true changes as possible (almost everything is strictly movement
of existing content). Note in particular that this commit does not fix
any indentation (we keep sect2 indentation for our new sect1). That
will happen in a later commit, after we're done splitting up content
from "The Autovacuum Daemon" into more subsections to improve its
readability.
XXX Open question: does it make more sense to move the sect1 to before
"Routine Vacuuming", or should it go after instead? There are arguments
for both.
Arguments for "before" (which is how it's done by this commit right
now):
"Before" gives greater prominence to the autovacuum scheduling tunables,
such as autovacuum_vacuum_scale_factor. These are the most important
individual tunables, which argues for putting them earlier than "Routine
Vacuuming".
Arguments for "after":
Although the discussion in "Routine Vacuuming" is rather involved, it is
arguably still introductory material that informs how the user will tune
autovacuum_vacuum_scale_factor. (Assuming that the user doesn't just go
by trial and error, which seems more likely in practice but not
necessarily the most useful working assumption for our purposes.)
---
doc/src/sgml/maintenance.sgml | 332 +++++++++++++++++-----------------
1 file changed, 166 insertions(+), 166 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 9cf9d030a..b9091e72c 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -59,6 +59,172 @@
pleasant and productive experience with the system.
</para>
+ <sect1 id="autovacuum">
+ <title>The Autovacuum Daemon</title>
+
+ <indexterm>
+ <primary>autovacuum</primary>
+ <secondary>general information</secondary>
+ </indexterm>
+ <para>
+ <productname>PostgreSQL</productname> has an optional but highly
+ recommended feature called <firstterm>autovacuum</firstterm>,
+ whose purpose is to automate the execution of
+ <command>VACUUM</command> and <command>ANALYZE</command> commands.
+ When enabled, autovacuum checks for
+ tables that have had a large number of inserted, updated or deleted
+ tuples. These checks use the statistics collection facility;
+ therefore, autovacuum cannot be used unless <xref
+ linkend="guc-track-counts"/> is set to <literal>true</literal>.
+ In the default configuration, autovacuuming is enabled and the related
+ configuration parameters are appropriately set.
+ </para>
+
+ <para>
+ The <quote>autovacuum daemon</quote> actually consists of multiple processes.
+ There is a persistent daemon process, called the
+ <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
+ <firstterm>autovacuum worker</firstterm> processes for all databases. The
+ launcher will distribute the work across time, attempting to start one
+ worker within each database every <xref linkend="guc-autovacuum-naptime"/>
+ seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
+ a new worker will be launched every
+ <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
+ A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
+ are allowed to run at the same time. If there are more than
+ <varname>autovacuum_max_workers</varname> databases to be processed,
+ the next database will be processed as soon as the first worker finishes.
+ Each worker process will check each table within its database and
+ execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
+ <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
+ autovacuum workers' activity.
+ </para>
+
+ <para>
+ If several large tables all become eligible for vacuuming in a short
+ amount of time, all autovacuum workers might become occupied with
+ vacuuming those tables for a long period. This would result
+ in other tables and databases not being vacuumed until a worker becomes
+ available. There is no limit on how many workers might be in a
+ single database, but workers do try to avoid repeating work that has
+ already been done by other workers. Note that the number of running
+ workers does not count towards <xref linkend="guc-max-connections"/> or
+ <xref linkend="guc-superuser-reserved-connections"/> limits.
+ </para>
+
+ <para>
+ Tables whose <structfield>relfrozenxid</structfield> value is more than
+ <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
+ vacuumed (this also applies to those tables whose freeze max age has
+ been modified via storage parameters; see below). Otherwise, if the
+ number of tuples obsoleted since the last
+ <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
+ table is vacuumed. The vacuum threshold is defined as:
+<programlisting>
+vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
+</programlisting>
+ where the vacuum base threshold is
+ <xref linkend="guc-autovacuum-vacuum-threshold"/>,
+ the vacuum scale factor is
+ <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
+ and the number of tuples is
+ <structname>pg_class</structname>.<structfield>reltuples</structfield>.
+ </para>
+
+ <para>
+ The table is also vacuumed if the number of tuples inserted since the last
+ vacuum has exceeded the defined insert threshold, which is defined as:
+<programlisting>
+vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
+</programlisting>
+ where the vacuum insert base threshold is
+ <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
+ and vacuum insert scale factor is
+ <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
+ Such vacuums may allow portions of the table to be marked as
+ <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
+ can reduce the work required in subsequent vacuums.
+ For tables which receive <command>INSERT</command> operations but no or
+ almost no <command>UPDATE</command>/<command>DELETE</command> operations,
+ it may be beneficial to lower the table's
+ <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
+ tuples to be frozen by earlier vacuums. The number of obsolete tuples and
+ the number of inserted tuples are obtained from the cumulative statistics system;
+ it is a semi-accurate count updated by each <command>UPDATE</command>,
+ <command>DELETE</command> and <command>INSERT</command> operation. (It is
+ only semi-accurate because some information might be lost under heavy
+ load.) If the <structfield>relfrozenxid</structfield> value of the table
+ is more than <varname>vacuum_freeze_table_age</varname> transactions old,
+ an aggressive vacuum is performed to freeze old tuples and advance
+ <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
+ since the last vacuum are scanned.
+ </para>
+
+ <para>
+ For analyze, a similar condition is used: the threshold, defined as:
+<programlisting>
+analyze threshold = analyze base threshold + analyze scale factor * number of tuples
+</programlisting>
+ is compared to the total number of tuples inserted, updated, or deleted
+ since the last <command>ANALYZE</command>.
+ </para>
+
+ <para>
+ Partitioned tables are not processed by autovacuum. Statistics
+ should be collected by running a manual <command>ANALYZE</command> when it is
+ first populated, and again whenever the distribution of data in its
+ partitions changes significantly.
+ </para>
+
+ <para>
+ Temporary tables cannot be accessed by autovacuum. Therefore,
+ appropriate vacuum and analyze operations should be performed via
+ session SQL commands.
+ </para>
+
+ <para>
+ The default thresholds and scale factors are taken from
+ <filename>postgresql.conf</filename>, but it is possible to override them
+ (and many other autovacuum control parameters) on a per-table basis; see
+ <xref linkend="sql-createtable-storage-parameters"/> for more information.
+ If a setting has been changed via a table's storage parameters, that value
+ is used when processing that table; otherwise the global settings are
+ used. See <xref linkend="runtime-config-autovacuum"/> for more details on
+ the global settings.
+ </para>
+
+ <para>
+ When multiple workers are running, the autovacuum cost delay parameters
+ (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
+ <quote>balanced</quote> among all the running workers, so that the
+ total I/O impact on the system is the same regardless of the number
+ of workers actually running. However, any workers processing tables whose
+ per-table <literal>autovacuum_vacuum_cost_delay</literal> or
+ <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
+ are not considered in the balancing algorithm.
+ </para>
+
+ <para>
+ Autovacuum workers generally don't block other commands. If a process
+ attempts to acquire a lock that conflicts with the
+ <literal>SHARE UPDATE EXCLUSIVE</literal> lock held by autovacuum, lock
+ acquisition will interrupt the autovacuum. For conflicting lock modes,
+ see <xref linkend="table-lock-compatibility"/>. However, if the autovacuum
+ is running to prevent transaction ID wraparound (i.e., the autovacuum query
+ name in the <structname>pg_stat_activity</structname> view ends with
+ <literal>(to prevent wraparound)</literal>), the autovacuum is not
+ automatically interrupted.
+ </para>
+
+ <warning>
+ <para>
+ Regularly running commands that acquire locks conflicting with a
+ <literal>SHARE UPDATE EXCLUSIVE</literal> lock (e.g., ANALYZE) can
+ effectively prevent autovacuums from ever completing.
+ </para>
+ </warning>
+ </sect1>
+
<sect1 id="routine-vacuuming">
<title>Routine Vacuuming</title>
@@ -749,172 +915,6 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect3>
</sect2>
-
- <sect2 id="autovacuum">
- <title>The Autovacuum Daemon</title>
-
- <indexterm>
- <primary>autovacuum</primary>
- <secondary>general information</secondary>
- </indexterm>
- <para>
- <productname>PostgreSQL</productname> has an optional but highly
- recommended feature called <firstterm>autovacuum</firstterm>,
- whose purpose is to automate the execution of
- <command>VACUUM</command> and <command>ANALYZE</command> commands.
- When enabled, autovacuum checks for
- tables that have had a large number of inserted, updated or deleted
- tuples. These checks use the statistics collection facility;
- therefore, autovacuum cannot be used unless <xref
- linkend="guc-track-counts"/> is set to <literal>true</literal>.
- In the default configuration, autovacuuming is enabled and the related
- configuration parameters are appropriately set.
- </para>
-
- <para>
- The <quote>autovacuum daemon</quote> actually consists of multiple processes.
- There is a persistent daemon process, called the
- <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
- <firstterm>autovacuum worker</firstterm> processes for all databases. The
- launcher will distribute the work across time, attempting to start one
- worker within each database every <xref linkend="guc-autovacuum-naptime"/>
- seconds. (Therefore, if the installation has <replaceable>N</replaceable> databases,
- a new worker will be launched every
- <varname>autovacuum_naptime</varname>/<replaceable>N</replaceable> seconds.)
- A maximum of <xref linkend="guc-autovacuum-max-workers"/> worker processes
- are allowed to run at the same time. If there are more than
- <varname>autovacuum_max_workers</varname> databases to be processed,
- the next database will be processed as soon as the first worker finishes.
- Each worker process will check each table within its database and
- execute <command>VACUUM</command> and/or <command>ANALYZE</command> as needed.
- <xref linkend="guc-log-autovacuum-min-duration"/> can be set to monitor
- autovacuum workers' activity.
- </para>
-
- <para>
- If several large tables all become eligible for vacuuming in a short
- amount of time, all autovacuum workers might become occupied with
- vacuuming those tables for a long period. This would result
- in other tables and databases not being vacuumed until a worker becomes
- available. There is no limit on how many workers might be in a
- single database, but workers do try to avoid repeating work that has
- already been done by other workers. Note that the number of running
- workers does not count towards <xref linkend="guc-max-connections"/> or
- <xref linkend="guc-superuser-reserved-connections"/> limits.
- </para>
-
- <para>
- Tables whose <structfield>relfrozenxid</structfield> value is more than
- <xref linkend="guc-autovacuum-freeze-max-age"/> transactions old are always
- vacuumed (this also applies to those tables whose freeze max age has
- been modified via storage parameters; see below). Otherwise, if the
- number of tuples obsoleted since the last
- <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
- table is vacuumed. The vacuum threshold is defined as:
-<programlisting>
-vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
-</programlisting>
- where the vacuum base threshold is
- <xref linkend="guc-autovacuum-vacuum-threshold"/>,
- the vacuum scale factor is
- <xref linkend="guc-autovacuum-vacuum-scale-factor"/>,
- and the number of tuples is
- <structname>pg_class</structname>.<structfield>reltuples</structfield>.
- </para>
-
- <para>
- The table is also vacuumed if the number of tuples inserted since the last
- vacuum has exceeded the defined insert threshold, which is defined as:
-<programlisting>
-vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
-</programlisting>
- where the vacuum insert base threshold is
- <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
- and vacuum insert scale factor is
- <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
- Such vacuums may allow portions of the table to be marked as
- <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
- can reduce the work required in subsequent vacuums.
- For tables which receive <command>INSERT</command> operations but no or
- almost no <command>UPDATE</command>/<command>DELETE</command> operations,
- it may be beneficial to lower the table's
- <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
- tuples to be frozen by earlier vacuums. The number of obsolete tuples and
- the number of inserted tuples are obtained from the cumulative statistics system;
- it is a semi-accurate count updated by each <command>UPDATE</command>,
- <command>DELETE</command> and <command>INSERT</command> operation. (It is
- only semi-accurate because some information might be lost under heavy
- load.) If the <structfield>relfrozenxid</structfield> value of the table
- is more than <varname>vacuum_freeze_table_age</varname> transactions old,
- an aggressive vacuum is performed to freeze old tuples and advance
- <structfield>relfrozenxid</structfield>; otherwise, only pages that have been modified
- since the last vacuum are scanned.
- </para>
-
- <para>
- For analyze, a similar condition is used: the threshold, defined as:
-<programlisting>
-analyze threshold = analyze base threshold + analyze scale factor * number of tuples
-</programlisting>
- is compared to the total number of tuples inserted, updated, or deleted
- since the last <command>ANALYZE</command>.
- </para>
-
- <para>
- Partitioned tables are not processed by autovacuum. Statistics
- should be collected by running a manual <command>ANALYZE</command> when it is
- first populated, and again whenever the distribution of data in its
- partitions changes significantly.
- </para>
-
- <para>
- Temporary tables cannot be accessed by autovacuum. Therefore,
- appropriate vacuum and analyze operations should be performed via
- session SQL commands.
- </para>
-
- <para>
- The default thresholds and scale factors are taken from
- <filename>postgresql.conf</filename>, but it is possible to override them
- (and many other autovacuum control parameters) on a per-table basis; see
- <xref linkend="sql-createtable-storage-parameters"/> for more information.
- If a setting has been changed via a table's storage parameters, that value
- is used when processing that table; otherwise the global settings are
- used. See <xref linkend="runtime-config-autovacuum"/> for more details on
- the global settings.
- </para>
-
- <para>
- When multiple workers are running, the autovacuum cost delay parameters
- (see <xref linkend="runtime-config-resource-vacuum-cost"/>) are
- <quote>balanced</quote> among all the running workers, so that the
- total I/O impact on the system is the same regardless of the number
- of workers actually running. However, any workers processing tables whose
- per-table <literal>autovacuum_vacuum_cost_delay</literal> or
- <literal>autovacuum_vacuum_cost_limit</literal> storage parameters have been set
- are not considered in the balancing algorithm.
- </para>
-
- <para>
- Autovacuum workers generally don't block other commands. If a process
- attempts to acquire a lock that conflicts with the
- <literal>SHARE UPDATE EXCLUSIVE</literal> lock held by autovacuum, lock
- acquisition will interrupt the autovacuum. For conflicting lock modes,
- see <xref linkend="table-lock-compatibility"/>. However, if the autovacuum
- is running to prevent transaction ID wraparound (i.e., the autovacuum query
- name in the <structname>pg_stat_activity</structname> view ends with
- <literal>(to prevent wraparound)</literal>), the autovacuum is not
- automatically interrupted.
- </para>
-
- <warning>
- <para>
- Regularly running commands that acquire locks conflicting with a
- <literal>SHARE UPDATE EXCLUSIVE</literal> lock (e.g., ANALYZE) can
- effectively prevent autovacuums from ever completing.
- </para>
- </warning>
- </sect2>
</sect1>
--
2.40.1
[application/octet-stream] v4-0009-Overhaul-freezing-and-wraparound-docs.patch (91.0K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/6-v4-0009-Overhaul-freezing-and-wraparound-docs.patch)
download | inline diff:
From 36666e87fe05ed24e413d1cd2f512bfacd65be38 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 13:04:13 -0700
Subject: [PATCH v4 9/9] Overhaul freezing and wraparound docs.
This is almost a complete rewrite. "Preventing Transaction ID
Wraparound Failures" becomes "Freezing to manage the transaction ID
space". This is follow-up work to commit 1de58df4, which added
page-level freezing to VACUUM.
The emphasis is now on the physical work of freezing pages. This flows
a little better than it otherwise would due to recent structural
cleanups to maintenance.sgml; discussion about freezing now immediately
follows discussion of cleanup of dead tuples. We still talk about the
problem of the system activating xidStopLimit protections in the same
section, but we use much less alarmist language about data corruption,
and are no longer overly concerned about the very worst case. We don't
rescind the recommendation that users recover from an xidStopLimit
outage by using single user mode, though that seems like something we
should aim to do in the near future.
There is no longer a separate sect3 to discuss MultiXactId related
issues. VACUUM now performs exactly the same processing steps when it
freezes a page, independent of the trigger condition.
Also move recommendation about setting autovacuum_freeze_min_age
reloption in append-only tables that was originally added by the
autovacuum_vacuum_insert_scale_factor commit over to "Routine
Vacuuming", where it now appears in the form of a "Tip" box.
Also describe the page-level freezing FPI optimization added by commit
1de58df4. This is expected to trigger the majority of all freezing with
many types of workloads.
Also move "table age" monitoring query to monitoring.sgml, though leave
behind a couple of forwarding links in maintenance.sgml's discussion of
freezing and relfrozenxid advancement.
---
doc/src/sgml/catalogs.sgml | 18 +-
doc/src/sgml/config.sgml | 79 +-
doc/src/sgml/logicaldecoding.sgml | 4 +-
doc/src/sgml/maintenance.sgml | 1062 ++++++++++++++++-----
doc/src/sgml/monitoring.sgml | 80 +-
doc/src/sgml/ref/create_table.sgml | 9 +-
doc/src/sgml/ref/prepare_transaction.sgml | 14 +-
doc/src/sgml/ref/vacuum.sgml | 14 +-
doc/src/sgml/ref/vacuumdb.sgml | 17 +-
doc/src/sgml/storage.sgml | 3 +-
doc/src/sgml/xact.sgml | 2 +-
11 files changed, 963 insertions(+), 339 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 524084055..7bb123e32 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2243,8 +2243,9 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<para>
All transaction IDs before this one have been replaced with a permanent
(<quote>frozen</quote>) transaction ID in this table. This is used to track
- whether the table needs to be vacuumed in order to prevent transaction
- ID wraparound or to allow <literal>pg_xact</literal> to be shrunk. Zero
+ whether the table needs an aggressive <command>VACUUM</command> (see
+ <xref linkend="vacuum-aggressive"/>) or an anti-wraparound autovacuum
+ (see <xref linkend="vacuum-antiwraparound-autovacuums"/>). Zero
(<symbol>InvalidTransactionId</symbol>) if the relation is not a table.
</para></entry>
</row>
@@ -2256,8 +2257,9 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<para>
All multixact IDs before this one have been replaced by a
transaction ID in this table. This is used to track
- whether the table needs to be vacuumed in order to prevent multixact ID
- wraparound or to allow <literal>pg_multixact</literal> to be shrunk. Zero
+ whether the table needs an aggressive <command>VACUUM</command> (see
+ <xref linkend="vacuum-aggressive"/>) or an anti-wraparound autovacuum
+ (see <xref linkend="vacuum-antiwraparound-autovacuums"/>). Zero
(<symbol>InvalidMultiXactId</symbol>) if the relation is not a table.
</para></entry>
</row>
@@ -3053,8 +3055,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<para>
All transaction IDs before this one have been replaced with a permanent
(<quote>frozen</quote>) transaction ID in this database. This is used to
- track whether the database needs to be vacuumed in order to prevent
- transaction ID wraparound or to allow <literal>pg_xact</literal> to be shrunk.
+ track whether the database allows <literal>pg_xact</literal> to be
+ shrunk (see <xref linkend="vacuum-truncate-xact-status"/>).
It is the minimum of the per-table
<link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relfrozenxid</structfield> values.
</para></entry>
@@ -3067,8 +3069,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<para>
All multixact IDs before this one have been replaced with a
transaction ID in this database. This is used to
- track whether the database needs to be vacuumed in order to prevent
- multixact ID wraparound or to allow <literal>pg_multixact</literal> to be shrunk.
+ track whether the database allows <literal>pg_multixact</literal> to be
+ shrunk (see <xref linkend="vacuum-truncate-xact-status"/>).
It is the minimum of the per-table
<link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relminmxid</structfield> values.
</para></entry>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 909a3f28c..0a94ff46a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2812,7 +2812,9 @@ include_dir 'conf.d'
<literal>1min</literal>) are only allowed because they may sometimes be
useful for testing. While a setting as high as <literal>60d</literal> is
allowed, please note that in many workloads extreme bloat or
- transaction ID wraparound may occur in much shorter time frames.
+ transaction ID exhaustion may occur in much shorter time frames
+ (see <xref linkend="vacuum-aggressive"/> and
+ <xref linkend="vacuum-xid-exhaustion"/>).
</para>
<para>
@@ -8358,9 +8360,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</para>
<para>
Note that even when this parameter is disabled, the system
- will launch autovacuum processes if necessary to
- prevent transaction ID wraparound. See <xref
- linkend="vacuum-for-wraparound"/> for more information.
+ will launch anti-wraparound autovacuums. See <xref
+ linkend="vacuum-antiwraparound-autovacuums"/> for more information.
</para>
</listitem>
</varlistentry>
@@ -8536,20 +8537,17 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
<para>
Specifies the maximum age (in transactions) that a table's
<structname>pg_class</structname>.<structfield>relfrozenxid</structfield> field can
- attain before a <command>VACUUM</command> operation is forced
- to prevent transaction ID wraparound within the table.
- Note that the system will launch autovacuum processes to
- prevent wraparound even when autovacuum is otherwise disabled.
+ attain before an anti-wraparound autovacuum is forced for the table.
+ Note that the system will launch anti-wraparound autovacuum
+ processes even when autovacuum is otherwise disabled.
</para>
<para>
- Vacuum also allows removal of old files from the
- <filename>pg_xact</filename> subdirectory, which is why the default
- is a relatively low 200 million transactions.
+ The default is 200 million transactions.
This parameter can only be set at server start, but the setting
can be reduced for individual tables by
changing table storage parameters.
- For more information see <xref linkend="vacuum-for-wraparound"/>.
+ For more information see <xref linkend="vacuum-antiwraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -8565,20 +8563,16 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
<para>
Specifies the maximum age (in multixacts) that a table's
<structname>pg_class</structname>.<structfield>relminmxid</structfield> field can
- attain before a <command>VACUUM</command> operation is forced to
- prevent multixact ID wraparound within the table.
- Note that the system will launch autovacuum processes to
- prevent wraparound even when autovacuum is otherwise disabled.
+ attain before an anti-wraparound autovacuum is forced for the table.
+ Note that the system will launch anti-wraparound autovacuum
+ processes even when autovacuum is otherwise disabled.
</para>
<para>
- Vacuuming multixacts also allows removal of old files from the
- <filename>pg_multixact/members</filename> and <filename>pg_multixact/offsets</filename>
- subdirectories, which is why the default is a relatively low
- 400 million multixacts.
+ The default is 400 million Multixact IDs.
This parameter can only be set at server start, but the setting can
be reduced for individual tables by changing table storage parameters.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ For more information see <xref linkend="vacuum-antiwraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -9282,10 +9276,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
set this value anywhere from zero to two billion, <command>VACUUM</command>
will silently limit the effective value to 95% of
<xref linkend="guc-autovacuum-freeze-max-age"/>, so that a
- periodic manual <command>VACUUM</command> has a chance to run before an
- anti-wraparound autovacuum is launched for the table. For more
- information see
- <xref linkend="vacuum-for-wraparound"/>.
+ standard autovacuum (or a manual <command>VACUUM</command>) has a
+ chance to run using <command>VACUUM</command>'s aggressive strategy
+ before an anti-wraparound autovacuum is launched for the table. For
+ more information see <xref linkend="vacuum-aggressive"/> and <xref
+ linkend="vacuum-antiwraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -9307,7 +9302,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
the value of <xref linkend="guc-autovacuum-freeze-max-age"/>, so
that there is not an unreasonably short time between forced
autovacuums. For more information see <xref
- linkend="vacuum-for-wraparound"/>.
+ linkend="vacuum-freezing-xid-space"/>.
</para>
</listitem>
</varlistentry>
@@ -9324,9 +9319,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
<structname>pg_class</structname>.<structfield>relfrozenxid</structfield>
field can attain before <command>VACUUM</command> takes
extraordinary measures to avoid system-wide transaction ID
- wraparound failure. This is <command>VACUUM</command>'s
- strategy of last resort. The failsafe typically triggers
- when an autovacuum to prevent transaction ID wraparound has
+ exhaustion (see <xref linkend="vacuum-xid-exhaustion"/>).
+ This is <command>VACUUM</command>'s strategy of last resort. The
+ failsafe typically triggers when an
+ <link linkend="vacuum-antiwraparound-autovacuums">anti-wraparound
+ autovacuum</link> has
already been running for some time, though it's possible for
the failsafe to trigger during any <command>VACUUM</command>.
</para>
@@ -9344,7 +9341,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
set this value anywhere from zero to 2.1 billion,
<command>VACUUM</command> will silently adjust the effective
value to no less than 105% of <xref
- linkend="guc-autovacuum-freeze-max-age"/>.
+ linkend="guc-autovacuum-freeze-max-age"/>. For more
+ information see <xref linkend="vacuum-xid-exhaustion"/>.
</para>
</listitem>
</varlistentry>
@@ -9366,9 +9364,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
Although users can set this value anywhere from zero to two billion,
<command>VACUUM</command> will silently limit the effective value to 95% of
<xref linkend="guc-autovacuum-multixact-freeze-max-age"/>, so that a
- periodic manual <command>VACUUM</command> has a chance to run before an
- anti-wraparound is launched for the table.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ standard autovacuum (or a manual <command>VACUUM</command>) has a
+ chance to run using <command>VACUUM</command>'s aggressive strategy
+ before an anti-wraparound autovacuum is launched for the table. For
+ more information see <xref linkend="vacuum-aggressive"/> and <xref
+ linkend="vacuum-antiwraparound-autovacuums"/>.
</para>
</listitem>
</varlistentry>
@@ -9389,7 +9389,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
the value of <xref linkend="guc-autovacuum-multixact-freeze-max-age"/>,
so that there is not an unreasonably short time between forced
autovacuums.
- For more information see <xref linkend="vacuum-for-multixact-wraparound"/>.
+ For more information see <xref linkend="vacuum-freezing-xid-space"/>.
</para>
</listitem>
</varlistentry>
@@ -9406,9 +9406,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
<structname>pg_class</structname>.<structfield>relminmxid</structfield>
field can attain before <command>VACUUM</command> takes
extraordinary measures to avoid system-wide multixact ID
- wraparound failure. This is <command>VACUUM</command>'s
- strategy of last resort. The failsafe typically triggers when
- an autovacuum to prevent transaction ID wraparound has already
+ exhaustion (see <xref linkend="vacuum-xid-exhaustion"/>).
+ This is <command>VACUUM</command>'s strategy of last resort. The
+ failsafe typically triggers when an
+ <link linkend="vacuum-antiwraparound-autovacuums">anti-wraparound
+ autovacuum</link> has already
been running for some time, though it's possible for the
failsafe to trigger during any <command>VACUUM</command>.
</para>
@@ -9422,7 +9424,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
this value anywhere from zero to 2.1 billion,
<command>VACUUM</command> will silently adjust the effective
value to no less than 105% of <xref
- linkend="guc-autovacuum-multixact-freeze-max-age"/>.
+ linkend="guc-autovacuum-multixact-freeze-max-age"/>. For more
+ information see <xref linkend="vacuum-xid-exhaustion"/>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index cbd3aa804..cc6499e36 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -352,8 +352,8 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU
even when there is no connection using them. This consumes storage
because neither required WAL nor required rows from the system catalogs
can be removed by <command>VACUUM</command> as long as they are required by a replication
- slot. In extreme cases this could cause the database to shut down to prevent
- transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ slot. In extreme cases this could cause the database to refuse to allocate new
+ transaction IDs (see <xref linkend="vacuum-xid-exhaustion"/>).
So if a slot is no longer required it should be dropped.
</para>
</caution>
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index f00442564..abdf01009 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -148,13 +148,8 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac
<xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
Such vacuums may allow portions of the table to be marked as
<firstterm>all visible</firstterm> and also allow tuples to be frozen, which
- can reduce the work required in subsequent vacuums.
- For tables which receive <command>INSERT</command> operations but no or
- almost no <command>UPDATE</command>/<command>DELETE</command> operations,
- it may be beneficial to lower the table's
- <xref linkend="reloption-autovacuum-freeze-min-age"/> as this may allow
- tuples to be frozen by earlier vacuums. The number of obsolete tuples and
- the number of inserted tuples are obtained from the cumulative statistics system;
+ can reduce the work required in subsequent vacuums. The number of obsolete tuples
+ and the number of inserted tuples are obtained from the cumulative statistics system;
it is a semi-accurate count updated by each <command>UPDATE</command>,
<command>DELETE</command> and <command>INSERT</command> operation. (It is
only semi-accurate because some information might be lost under heavy
@@ -211,10 +206,11 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<literal>SHARE UPDATE EXCLUSIVE</literal> lock held by autovacuum, lock
acquisition will interrupt the autovacuum. For conflicting lock modes,
see <xref linkend="table-lock-compatibility"/>. However, if the autovacuum
- is running to prevent transaction ID wraparound (i.e., the autovacuum query
- name in the <structname>pg_stat_activity</structname> view ends with
+ is an anti-wraparound autovacuum (i.e., the autovacuum query name in the
+ <structname>pg_stat_activity</structname> view ends with
<literal>(to prevent wraparound)</literal>), the autovacuum is not
- automatically interrupted.
+ automatically interrupted. See <xref linkend="vacuum-antiwraparound-autovacuums"/>
+ for more details on anti-wraparound autovacuums.
</para>
<warning>
@@ -272,15 +268,21 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ <simpara>To maintain the system's ability to allocate transaction IDs
+ through freezing.</simpara>
</listitem>
<listitem>
<simpara>To update the visibility map, which speeds
up <link linkend="indexes-index-only-scans">index-only
- scans</link>.</simpara>
+ scans</link>, and helps the next <command>VACUUM</command>
+ operation avoid needlessly scanning already-frozen pages.</simpara>
+ </listitem>
+
+ <listitem>
+ <simpara>To enable truncation of obsolescent transaction status
+ information in structures such as <filename>pg_xact</filename> for the
+ entire cluster.</simpara>
</listitem>
<listitem>
@@ -477,303 +479,756 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</warning>
</sect2>
- <sect2 id="vacuum-for-wraparound">
- <title>Preventing Transaction ID Wraparound Failures</title>
-
- <indexterm zone="vacuum-for-wraparound">
- <primary>transaction ID</primary>
- <secondary>wraparound</secondary>
- </indexterm>
+ <sect2 id="vacuum-freezing-xid-space">
+ <title>Freezing to manage the transaction ID space</title>
<indexterm>
- <primary>wraparound</primary>
- <secondary>of transaction IDs</secondary>
+ <primary>Freezing</primary>
+ <secondary>of transaction IDs and Multixact IDs</secondary>
</indexterm>
<para>
- <productname>PostgreSQL</productname>'s <link
- linkend="mvcc-intro">MVCC</link> transaction semantics depend on
- being able to compare <glossterm linkend="glossary-xid">transaction
- ID numbers (<acronym>XID</acronym>)</glossterm> to determine
- whether or not the row is visible to each query's MVCC snapshot
- (see <xref linkend="interpreting-xid-stamps"/>). But since
- on-disk storage of transaction IDs in heap pages uses a truncated
- 32-bit representation to save space (rather than the full 64-bit
- representation), it is necessary to vacuum every table in every
- database <emphasis>at least</emphasis> once every two billion
- transactions (though far more frequent vacuuming is typical).
+ <command>VACUUM</command> often marks some of the pages that it scans
+ <emphasis>frozen</emphasis>, indicating that all eligible rows on the page
+ were inserted by a transaction that committed sufficiently far in the past
+ that the effects of the inserting transaction are certain to be visible to
+ all current and future transactions. The specific transaction ID number
+ (<acronym>XID</acronym>) stored in a frozen heap row's
+ <structfield>xmin</structfield> field is no longer needed to determine its
+ visibility. Furthermore, when a row undergoing freezing has an XID set in
+ its <structfield>xmax</structfield> field (e.g., an XID left behind by an
+ earlier <command>SELECT FOR UPDATE</command> row locker), the
+ <structfield>xmax</structfield> field's XID is usually also removed.
</para>
<para>
- <xref linkend="guc-vacuum-freeze-min-age"/>
- controls how old an XID value has to be before rows bearing that XID will be
- frozen. Increasing this setting may avoid unnecessary work if the
- rows that would otherwise be frozen will soon be modified again,
- but decreasing this setting increases
- the number of transactions that can elapse before the table must be
- vacuumed again.
+ Once frozen, heap pages are <quote>self-contained</quote>. Every query
+ can read all of the page's rows in a way that assumes that the inserting
+ transaction committed and is visible to its <acronym>MVCC</acronym>
+ snapshot. No query will ever have to consult external transaction status
+ metadata to interpret the page's contents, either. In particular,
+ <filename>pg_xact</filename> transaction XID commit/abort status lookups
+ won't occur during query execution.
</para>
<para>
- <command>VACUUM</command> uses the <link linkend="storage-vm">visibility map</link>
- to determine which pages of a table must be scanned. Normally, it
- will skip pages that don't have any dead row versions even if those pages
- might still have row versions with old XID values. Therefore, normal
- <command>VACUUM</command>s won't always freeze every old row version in the table.
- When that happens, <command>VACUUM</command> will eventually need to perform an
- <firstterm>aggressive vacuum</firstterm>, which will freeze all eligible unfrozen
- XID and MXID values, including those from all-visible but not all-frozen pages.
- In practice most tables require periodic aggressive vacuuming.
- <xref linkend="guc-vacuum-freeze-table-age"/>
- controls when <command>VACUUM</command> does that: all-visible but not all-frozen
- pages are scanned if the number of transactions that have passed since the
- last such scan is greater than <varname>vacuum_freeze_table_age</varname> minus
- <varname>vacuum_freeze_min_age</varname>. Setting
- <varname>vacuum_freeze_table_age</varname> to 0 forces <command>VACUUM</command> to
- always use its aggressive strategy.
+ Freezing is a <acronym>WAL</acronym>-logged operation, so when
+ <command>VACUUM</command> freezes a heap page, any copy of the page
+ located on a physical replication standby server will itself be
+ <quote>frozen</quote> shortly thereafter (when the relevant
+ <literal>FREEZE_PAGE</literal> <acronym>WAL</acronym> record is replayed
+ on the standby). Queries that run on physical replication standbys avoid
+ <filename>pg_xact</filename> lookups when reading from frozen pages, just
+ like queries that run on the primary server
+ <footnote>
+ <para>
+ In this regard, freezing is unlike setting transaction status
+ <quote>hint bits</quote> in tuple headers: setting hint bits doesn't
+ usually need to be <acronym>WAL</acronym>-logged, and can take place on
+ physical replication standby servers without input from the primary
+ server. Hint bits exist to allow query execution to avoid repeated
+ <filename>pg_xact</filename> lookups for the same tuples, strictly as an
+ optimization. On the other hand, freezing exists because the system
+ needs to reliably remove <filename>pg_xact</filename> dependencies from
+ individual tuples.
+ </para>
+ </footnote>.
</para>
<para>
- The maximum time that a table can go unvacuumed is two billion
- transactions minus the <varname>vacuum_freeze_min_age</varname> value at
- the time of the last aggressive vacuum. If it were to go
- unvacuumed for longer than
- that, data loss could result. To ensure that this does not happen,
- autovacuum is invoked on any table that might contain unfrozen rows with
- XIDs older than the age specified by the configuration parameter <xref
- linkend="guc-autovacuum-freeze-max-age"/>. (This will happen even if
- autovacuum is disabled.)
+ <command>VACUUM</command> generally postpones some freezing work as an
+ optimization, but <command>VACUUM</command> cannot delay freezing forever.
+ Since on-disk storage of transaction IDs in heap row headers uses a
+ truncated partial 32-bit representation to save space (rather than the
+ full 64-bit representation used in other contexts), it plays a crucial
+ role in enabling <link linkend="vacuum-aggressive">management of the XID
+ address space</link> by <command>VACUUM</command>. If, for whatever
+ reason, <command>VACUUM</command> is unable to freeze older XIDs on behalf
+ of an application that continues to require XID allocations, the system
+ will eventually <link linkend="vacuum-xid-exhaustion">refuse to allocate
+ transaction IDs</link> due to transaction ID exhaustion (though this is
+ unlikely to occur unless autovacuum is configured incorrectly).
</para>
<para>
- This implies that if a table is not otherwise vacuumed,
- autovacuum will be invoked on it approximately once every
- <varname>autovacuum_freeze_max_age</varname> minus
- <varname>vacuum_freeze_min_age</varname> transactions.
- For tables that are regularly vacuumed for space reclamation purposes,
- this is of little importance. However, for static tables
- (including tables that receive inserts, but no updates or deletes),
- there is no need to vacuum for space reclamation, so it can
- be useful to try to maximize the interval between forced autovacuums
- on very large static tables. Obviously one can do this either by
- increasing <varname>autovacuum_freeze_max_age</varname> or decreasing
- <varname>vacuum_freeze_min_age</varname>.
+ <xref linkend="guc-vacuum-freeze-min-age"/> controls when freezing takes
+ place. When <command>VACUUM</command> scans a heap page containing even
+ one XID that has already attained an age exceeding this value, the page is
+ frozen.
+ </para>
+
+ <indexterm>
+ <primary>Multixact ID</primary>
+ <secondary>Freezing of</secondary>
+ </indexterm>
+
+ <para>
+ <firstterm>Multixact IDs</firstterm> support row locking by multiple
+ transactions. Since there is only limited space in a <link
+ linkend="storage-tuple-layout">heap tuple header</link> to store lock
+ information, that information is encoded as a <quote>multiple transaction
+ ID</quote>, or Multixact ID for short, whenever there is more than one
+ transaction concurrently locking a row. Information about which
+ transaction IDs are included in any particular Multixact ID is stored
+ separately in <filename>pg_multixact</filename>. Only the Multixact ID
+ itself (a 32-bit integer) appears in the tuple's
+ <structfield>xmax</structfield> field. This creates a dependency on
+ external Multixact ID transaction status information. This is similar to
+ the dependency ordinary unfrozen XIDs have on commit status information
+ from <filename>pg_xact</filename>. <command>VACUUM</command> must
+ therefore occasionally remove Multixact IDs from tuples during freezing.
</para>
<para>
- The effective maximum for <varname>vacuum_freeze_table_age</varname> is 0.95 *
- <varname>autovacuum_freeze_max_age</varname>; a setting higher than that will be
- capped to the maximum. A value higher than
- <varname>autovacuum_freeze_max_age</varname> wouldn't make sense because an
- anti-wraparound autovacuum would be triggered at that point anyway, and
- the 0.95 multiplier leaves some breathing room to run a manual
- <command>VACUUM</command> before that happens. As a rule of thumb,
- <command>vacuum_freeze_table_age</command> should be set to a value somewhat
- below <varname>autovacuum_freeze_max_age</varname>, leaving enough gap so that
- a regularly scheduled <command>VACUUM</command> or an autovacuum triggered by
- normal delete and update activity is run in that window. Setting it too
- close could lead to anti-wraparound autovacuums, even though the table
- was recently vacuumed to reclaim space, whereas lower values lead to more
- frequent aggressive vacuuming.
+ <xref linkend="guc-vacuum-multixact-freeze-min-age"/> also controls when
+ freezing takes place. It is analogous to
+ <varname>vacuum_freeze_min_age</varname>, but <quote>age</quote> is
+ expressed in Multixact ID units. Lowering
+ <varname>vacuum_multixact_freeze_min_age</varname>
+ <emphasis>forces</emphasis> <command>VACUUM</command> to process
+ <structfield>xmax</structfield> fields with a Multixact ID in cases where
+ it would otherwise postpone the work of processing
+ <structfield>xmax</structfield> until the next <command>VACUUM</command>
+ <footnote>
+ <para>
+ <quote>Freezing</quote> of <structfield>xmax</structfield> fields
+ (whether they contain an XID or a Multixact ID) generally means clearing
+ <structfield>xmax</structfield> from a tuple header.
+ <command>VACUUM</command> may occasionally encounter an individual
+ Multixact ID that must be removed to advance the table's
+ <structfield>relminmxid</structfield> by the required amount, which can
+ only be processed by generating a replacement Multixact ID (containing
+ just the non-removable subset of member XIDs from the original Multixact
+ ID), and then setting <structfield>xmax</structfield> to the
+ new/replacement Multixact ID value.
+ </para>
+ </footnote>. The setting generally doesn't significantly influence the
+ total number of pages <command>VACUUM</command> freezes, even in tables
+ containing many Multixact IDs. This is because <command>VACUUM</command>
+ generally prefers proactive processing for most individual
+ <structfield>xmax</structfield> fields that contain a Multixact ID (eager
+ proactive processing is typically cheaper).
</para>
<para>
- The sole disadvantage of increasing <varname>autovacuum_freeze_max_age</varname>
- (and <varname>vacuum_freeze_table_age</varname> along with it) is that
- the <filename>pg_xact</filename> and <filename>pg_commit_ts</filename>
- subdirectories of the database cluster will take more space, because it
- must store the commit status and (if <varname>track_commit_timestamp</varname> is
- enabled) timestamp of all transactions back to
- the <varname>autovacuum_freeze_max_age</varname> horizon. The commit status uses
- two bits per transaction, so if
- <varname>autovacuum_freeze_max_age</varname> is set to its maximum allowed value
- of two billion, <filename>pg_xact</filename> can be expected to grow to about half
- a gigabyte and <filename>pg_commit_ts</filename> to about 20GB. If this
- is trivial compared to your total database size,
- setting <varname>autovacuum_freeze_max_age</varname> to its maximum allowed value
- is recommended. Otherwise, set it depending on what you are willing to
- allow for <filename>pg_xact</filename> and <filename>pg_commit_ts</filename> storage.
- (The default, 200 million transactions, translates to about 50MB
- of <filename>pg_xact</filename> storage and about 2GB of <filename>pg_commit_ts</filename>
- storage.)
+ Managing the added <acronym>WAL</acronym> volume from freezing over time
+ is a vital consideration for <command>VACUUM</command>. It is why
+ <command>VACUUM</command> doesn't just freeze every eligible tuple at the
+ earliest opportunity: the <acronym>WAL</acronym> written to freeze a
+ page's tuples is wasted in cases where the resulting frozen tuples are
+ soon deleted or updated anyway. It's also why <command>VACUUM</command>
+ <emphasis>will</emphasis> freeze all eligible tuples from a heap page once
+ the decision to freeze at least one tuple is taken: at that point, the
+ added cost of freezing all eligible tuples eagerly (measured in
+ <quote>extra bytes of <acronym>WAL</acronym> written</quote>) is far lower
+ than the probable cost of deferring freezing until a future
+ <command>VACUUM</command> operation against the same table. Furthermore,
+ once the page is frozen, it can generally be <link
+ linkend="vacuum-for-visibility-map">marked as all-frozen within the
+ visibility map</link> immediately afterwards.
</para>
- <para>
- One disadvantage of decreasing <varname>vacuum_freeze_min_age</varname> is that
- it might cause <command>VACUUM</command> to do useless work: freezing a row
- version is a waste of time if the row is modified
- soon thereafter (causing it to acquire a new XID). So the setting should
- be large enough that rows are not frozen until they are unlikely to change
- any more.
- </para>
+ <note>
+ <para>
+ In <productname>PostgreSQL</productname> versions before 16,
+ <command>VACUUM</command> triggered freezing at the level of individual
+ <structfield>xmin</structfield> and <structfield>xmax</structfield>
+ fields. Freezing only affected the exact XIDs that had already attained
+ an age of <varname>vacuum_freeze_min_age</varname> or greater.
+ </para>
+ </note>
<para>
- To track the age of the oldest unfrozen XIDs in a database,
- <command>VACUUM</command> stores XID
- statistics in the system tables <structname>pg_class</structname> and
- <structname>pg_database</structname>. In particular,
- the <structfield>relfrozenxid</structfield> column of a table's
- <structname>pg_class</structname> row contains the oldest remaining unfrozen
- XID at the end of the most recent <command>VACUUM</command> that successfully
- advanced <structfield>relfrozenxid</structfield> (typically the most recent
- aggressive VACUUM). Similarly, the
- <structfield>datfrozenxid</structfield> column of a database's
- <structname>pg_database</structname> row is a lower bound on the unfrozen XIDs
- appearing in that database — it is just the minimum of the
- per-table <structfield>relfrozenxid</structfield> values within the database.
- A convenient way to
- examine this information is to execute queries such as:
-
-<programlisting>
-SELECT c.oid::regclass as table_name,
- greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age
-FROM pg_class c
-LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
-WHERE c.relkind IN ('r', 'm');
-
-SELECT datname, age(datfrozenxid) FROM pg_database;
-</programlisting>
-
- The <literal>age</literal> column measures the number of transactions from the
- cutoff XID to the current transaction's XID.
+ <command>VACUUM</command> also triggers the freezing of a page in cases
+ where it already proved necessary to write out a full page image
+ (<acronym>FPI</acronym>) as part of a <acronym>WAL</acronym> record
+ describing how dead tuples were removed <footnote>
+ <para>
+ Actually, the <quote>freeze on an <acronym>FPI</acronym> write</quote>
+ mechanism isn't just used when <command>VACUUM</command> needs to
+ generate an <acronym>FPI</acronym> (as torn page protection) for
+ inclusion in a <acronym>WAL</acronym> record describing how dead tuples
+ were removed. The <acronym>FPI</acronym> mechanism also triggers when
+ hint bits are set by <command>VACUUM</command>, if and only if it
+ necessitates writing an <acronym>FPI</acronym>. The need to write a
+ <acronym>WAL</acronym> record to set hint bits only arises when
+ <xref linkend="guc-wal-log-hints"/> is enabled in
+ <filename>postgresql.conf</filename>, or when data checksums were
+ enabled when the cluster was initialized with <xref linkend="app-initdb"/>.
+ </para>
+ </footnote> (see <xref linkend="wal-reliability"/> for background
+ information about how <acronym>FPI</acronym>s provide torn page
+ protection). This <quote>freeze on an <acronym>FPI</acronym>
+ write</quote> batching mechanism avoids an expected additional
+ <acronym>FPI</acronym> for the same page later on (this is the probable
+ outcome of lazily deferring freezing until <varname>vacuum_freeze_min_age</varname>
+ forces it). In effect, <command>VACUUM</command> generates slightly more
+ <acronym>WAL</acronym> in the short term with the aim of ultimately
+ needing to generate much less <acronym>WAL</acronym> in the long term.
</para>
<tip>
<para>
- When the <command>VACUUM</command> command's <literal>VERBOSE</literal>
- parameter is specified, <command>VACUUM</command> prints various
- statistics about the table. This includes information about how
- <structfield>relfrozenxid</structfield> and
- <structfield>relminmxid</structfield> advanced, and the number of
- newly frozen pages. The same details appear in the server log when
- autovacuum logging (controlled by <xref
- linkend="guc-log-autovacuum-min-duration"/>) reports on a
- <command>VACUUM</command> operation executed by autovacuum.
+ For tables that receive <command>INSERT</command> operations, but few or
+ no <command>UPDATE</command>/<command>DELETE</command> operations, it
+ might be beneficial to lower <xref linkend="reloption-autovacuum-freeze-min-age"/>
+ for the table. This makes <command>VACUUM</command> freeze the table's
+ pages <quote>eagerly</quote> during earlier autovacuums triggered by
+ <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>, which
+ improves performance stability for some workloads.
</para>
</tip>
- <para>
- <command>VACUUM</command> normally only scans pages that have been modified
- since the last vacuum, but <structfield>relfrozenxid</structfield> can only be
- advanced when every page of the table
- that might contain unfrozen XIDs is scanned. This happens when
- <structfield>relfrozenxid</structfield> is more than
- <varname>vacuum_freeze_table_age</varname> transactions old, when
- <command>VACUUM</command>'s <literal>FREEZE</literal> option is used, or when all
- pages that are not already all-frozen happen to
- require vacuuming to remove dead row versions. When <command>VACUUM</command>
- scans every page in the table that is not already all-frozen, it should
- set <literal>age(relfrozenxid)</literal> to a value just a little more than the
- <varname>vacuum_freeze_min_age</varname> setting
- that was used (more by the number of transactions started since the
- <command>VACUUM</command> started). <command>VACUUM</command>
- will set <structfield>relfrozenxid</structfield> to the oldest XID
- that remains in the table, so it's possible that the final value
- will be much more recent than strictly required.
- If no <structfield>relfrozenxid</structfield>-advancing
- <command>VACUUM</command> is issued on the table until
- <varname>autovacuum_freeze_max_age</varname> is reached, an autovacuum will soon
- be forced for the table.
- </para>
+ <sect3 id="vacuum-aggressive">
+ <title>Aggressive <command>VACUUM</command></title>
- <para>
- If for some reason autovacuum fails to clear old XIDs from a table, the
- system will begin to emit warning messages like this when the database's
- oldest XIDs reach forty million transactions from the wraparound point:
+ <indexterm zone="vacuum-aggressive">
+ <primary>transaction ID</primary>
+ <secondary>wraparound</secondary>
+ </indexterm>
+
+ <indexterm>
+ <primary>wraparound</primary>
+ <secondary>of transaction IDs and Multixact IDs</secondary>
+ </indexterm>
+
+ <para>
+ As noted already, freezing doesn't just allow queries to avoid lookups of
+ subsidiary transaction status information in structures such as
+ <filename>pg_xact</filename>. Freezing also plays a crucial role in
+ enabling transaction ID address space management by
+ <command>VACUUM</command> (and autovacuum). <command>VACUUM</command>
+ maintains information about the oldest unfrozen XID that remains in the
+ table when it uses its <firstterm>aggressive strategy</firstterm>.
+ </para>
+
+ <para>
+ Aggressive <command>VACUUM</command> updates the table's <link
+ linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relfrozenxid</structfield>
+ to whatever XID was the oldest observed XID that
+ <command>VACUUM</command> <emphasis>didn't</emphasis> freeze still
+ remaining at the end of processing. The table's
+ <structfield>relfrozenxid</structfield> <quote>advances</quote> by a
+ certain number of XIDs (relative to the previous value set during the last
+ aggressive <command>VACUUM</command>) as progress on freezing the oldest
+ pages in the table permits. Aggressive <command>VACUUM</command> will
+ occasionally need to advance the whole database's <link
+ linkend="catalog-pg-database"><structname>pg_database</structname></link>.<structfield>datfrozenxid</structfield>
+ afterwards, too — this is the minimum of the per-table
+ <structfield>relfrozenxid</structfield> values (i.e., the earliest
+ <structfield>relfrozenxid</structfield>) within the database.
+ </para>
+
+ <para>
+ Aggressive <command>VACUUM</command> may need to perform significant
+ amounts of <quote>catch-up</quote> freezing missed by earlier
+ non-aggressive <command>VACUUM</command>s, because non-aggressive
+ <command>VACUUM</command> sometimes allows unfrozen pages to build up.
+ </para>
+
+ <para>
+ Over time, aggressive autovacuuming has two beneficial effects on the
+ system as a whole:
+
+ <orderedlist>
+ <listitem>
+ <simpara>It keeps track of the oldest remaining unfrozen transaction ID
+ in the entire <glossterm linkend="glossary-db-cluster">database
+ cluster</glossterm> (i.e., the oldest transaction ID across every
+ table in every database).</simpara>
+ </listitem>
+ <listitem>
+ <simpara>It avoids a cluster-wide oldest unfrozen transaction ID that
+ is <quote>too old</quote>.</simpara>
+ </listitem>
+ </orderedlist>
+ </para>
+
+ <para>
+ The maximum XID age that the system can tolerate (i.e., the maximum
+ <quote>distance</quote> between the oldest unfrozen transaction ID in any
+ table in the database cluster and the next unallocated transaction ID) is
+ about 2.1 billion transaction IDs. This <quote>maximum XID age</quote>
+ invariant makes it fundamentally impossible to postpone aggressive
+ <command>VACUUM</command>s (and freezing) forever. While there is no
+ simple formula for determining an oldest XID <quote>age</quote> for
+ database administrators to target, the invariant imposes a 2.1 billion
+ XID age hard limit — so there <emphasis>is</emphasis> a clear point
+ at which unfrozen XIDs should <emphasis>always</emphasis> be considered
+ <quote>too old</quote>, regardless of individual application requirements
+ or workload characteristics. If the hard limit is reached, the system
+ experiences <link linkend="vacuum-xid-exhaustion">transaction ID
+ exhaustion</link>, which temporarily prevents the allocation of new
+ permanent transaction IDs. The system will only regain the ability to
+ allocate new transaction IDs when <command>VACUUM</command> succeeds in
+ advancing the oldest <literal>datfrozenxid</literal> in the cluster
+ (following an aggressive <command>VACUUM</command> that runs to
+ completion against the table with the oldest
+ <structfield>relfrozenxid</structfield>).
+ </para>
+
+ <para>
+ Aggressive <command>VACUUM</command> also maintains the <link
+ linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relminmxid</structfield> and
+ <link linkend="catalog-pg-database"><structname>pg_database</structname></link>.<structfield>datminmxid</structfield>
+ fields. These are needed to track the oldest Multixact ID in the table
+ and database, respectively. There are analogous rules, driven by
+ analogous considerations about managing the Multixact ID space. This
+ doesn't usually affect aggressive vacuuming requirements to a noticeable
+ degree, but can in databases that consume more Multixact IDs than
+ transaction IDs.
+ </para>
+
+ <caution>
+ <para>
+ <command>VACUUM</command> may not always freeze tuple
+ <structfield>xmin</structfield> XIDs that have reached
+ <varname>vacuum_freeze_min_age</varname> in age. The basic eligibility
+ criteria for freezing is the same as the criteria that determines if a
+ deleted tuple is safe for <command>VACUUM</command> to remove: the
+ XID-based <literal>removable cutoff</literal> (this is one of the
+ details that appears in the server log's reports on autovacuum
+ <footnote id="vacuum-autovacuum-log">
+ <para>
+ Autovacuum's log reports appear in the server log for autovacuums
+ whose <command>VACUUM</command> takes longer than a threshold
+ controlled by <xref linkend="guc-log-autovacuum-min-duration"/>.
+ Manual <command>VACUUM</command>s output the same details as
+ <literal>INFO</literal> messages when the <command>VACUUM</command>
+ command's <literal>VERBOSE</literal> option is used (note that manual
+ <command>VACUUM</command>s never generate reports in the server log).
+ </para>
+ </footnote>).
+ </para>
+ <para>
+ In extreme cases, a long-running transaction can hold back every
+ <command>VACUUM</command>'s <literal>removable cutoff</literal> for so
+ long that the system experiences
+ <link linkend="vacuum-xid-exhaustion">transaction ID exhaustion</link>.
+ See <xref linkend="monitoring-table-age"/> for details on how to monitor
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> age to avoid
+ transaction ID/Multixact ID exhaustion.
+ </para>
+ <para>
+ These issues can be debugged by following autovacuum log reports from
+ the server log over time: the log reports will include information about
+ the age of each <command>VACUUM</command>'s <literal>removable
+ cutoff</literal> at the point the <command>VACUUM</command> ended.
+ It may be useful to correlate the use of a cutoff with an excessively
+ high age with application-level problems such as long-running
+ transactions.
+ </para>
+ </caution>
+
+ <para>
+ The 2.1 billion XIDs <quote>maximum XID age</quote> invariant must be
+ preserved because transaction IDs stored in <link
+ linkend="storage-tuple-layout">heap tuple headers</link> use a truncated
+ 32-bit representation (rather than the full 64-bit representation used in
+ other contexts). Since all unfrozen transaction IDs from heap tuple
+ headers <emphasis>must</emphasis> be from the same transaction ID epoch
+ (or from a space in the 64-bit representation that spans two adjoining
+ transaction ID epochs), there isn't any need to include a separate epoch
+ field in each tuple header (see <xref linkend="interpreting-xid-stamps"/>
+ for further details). This scheme requires much less on-disk storage
+ space than a design that stores full 64-bit XIDs (consisting of a 32-bit
+ epoch and a 32-bit partial XID) in heap tuple headers. On the other
+ hand, it constrains the system's ability to allocate new XIDs in the
+ worst case scenario where transaction ID exhaustion occurs.
+ </para>
+
+ <para>
+ There is only one <emphasis>major</emphasis> behavioral difference
+ between aggressive <command>VACUUM</command> and non-aggressive
+ <command>VACUUM</command>: non-aggressive <command>VACUUM</command> skips
+ pages marked as all-visible using the visibility map, whereas aggressive
+ <command>VACUUM</command> only skips the subset of pages that are both
+ all-visible <emphasis>and</emphasis> all-frozen. In other words, pages
+ that are <emphasis>just</emphasis> all-visible at the beginning of an
+ aggressive <command>VACUUM</command> must be scanned, not skipped.
+ Scanning existing all-visible pages is necessary to determine the oldest
+ unfrozen XID that will remain in the table at the end of an aggressive
+ <command>VACUUM</command>.
+ </para>
+
+ <note>
+ <para>
+ In practice, most tables require periodic aggressive vacuuming.
+ However, some individual non-aggressive <command>VACUUM</command>
+ operations can advance the table's
+ <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield>.
+ </para>
+ <para>
+ This happens whenever a non-aggressive <command>VACUUM</command> notices
+ that it is safe without incurring any added cost from scanning
+ <quote>extra</quote> pages. It is most common in small, frequently
+ modified tables.
+ </para>
+ </note>
+
+ <para>
+ Non-aggressive <command>VACUUM</command>s can sometimes overlook older
+ XIDs from existing all-visible pages (due to their policy of always
+ skipping all-visible pages). Over time, this can even lead to a
+ significant build-up of unfrozen pages in one table (accumulated
+ all-visible pages that remain unfrozen). When that happens, it is
+ inevitable that an aggressive <command>VACUUM</command> will eventually
+ need to perform <quote>catch-up</quote> freezing that clears the table's
+ backlog of unfrozen pages.
+ </para>
+
+ <para>
+ There is also one <emphasis>minor</emphasis> behavioral difference
+ between aggressive <command>VACUUM</command> and non-aggressive
+ <command>VACUUM</command>: only aggressive <command>VACUUM</command> is
+ required to sometimes wait for a page-level cleanup lock when a page is
+ scanned and observed to contain transaction IDs/Multixact IDs that
+ <emphasis>must</emphasis> be frozen. This difference exists because
+ aggressive <command>VACUUM</command> is strictly required to advance
+ <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield> to
+ <emphasis>sufficiently</emphasis> recent values
+ <footnote>
+ <para>
+ Aggressive <command>VACUUM</command> is (somewhat arbitrarily) required
+ to freeze all pages containing transaction IDs older than
+ <varname>vacuum_freeze_min_age</varname> and/or Multixact ID values
+ older than <varname>vacuum_multixact_freeze_min_age</varname>, at a
+ minimum.
+ </para>
+ </footnote>. The behavior can lead to occasional waits for a conflicting
+ buffer pin to be released by another backend. These waits are
+ imperceptible and harmless most of the time. In extreme cases there can
+ be extended waits, which can be observed under the
+ <literal>BufferPin</literal> wait event in the
+ <structname>pg_stat_activity</structname> view. See <xref
+ linkend="wait-event-table"/>.
+ </para>
+
+ <note>
+ <para>
+ <quote>Catch-up</quote> freezing is not caused by any difference in how
+ <varname>vacuum_freeze_min_age</varname> is applied by each type of
+ <command>VACUUM</command>. It is an indirect result of
+ <varname>vacuum_freeze_min_age</varname> only being applied to those
+ pages that <command>VACUUM</command> scans (and cleanup locks) in the
+ first place. Therefore, it can be difficult to tune
+ <varname>vacuum_freeze_min_age</varname>, especially for tables that
+ receive frequent non-aggressive <command>VACUUM</command>s and
+ infrequent aggressive <command>VACUUM</command>s.
+ </para>
+ </note>
+
+ <tip>
+ <para>
+ Autovacuum server log reports <footnoteref
+ linkend="vacuum-autovacuum-log"/> show how many transaction IDs
+ <structfield>relfrozenxid</structfield> advanced by (if at all), and how
+ many Multixact IDs <structfield>relminmxid</structfield> advanced by (if
+ at all).
+ </para>
+ <para>
+ The number of pages frozen, and the number of pages scanned (i.e., the
+ number of pages processed because they could <emphasis>not</emphasis>
+ skipped using the visibility map) are also shown. This can provide
+ useful guidance when tuning freezing-related settings, particularly
+ <varname>vacuum_freeze_table_age</varname> and
+ <varname>vacuum_freeze_min_age</varname>.
+ </para>
+ </tip>
+
+ <para>
+ <xref linkend="guc-vacuum-freeze-table-age"/> controls when
+ <command>VACUUM</command> uses its aggressive strategy. If
+ <literal>age(relfrozenxid)</literal> exceeds
+ <varname>vacuum_freeze_table_age</varname> at the start of
+ <command>VACUUM</command>, <command>VACUUM</command> will employ its
+ aggressive strategy; otherwise, its standard non-aggressive strategy is
+ employed. Setting <varname>vacuum_freeze_table_age</varname> to 0 forces
+ <command>VACUUM</command> to always use its aggressive strategy.
+ </para>
+
+ <para>
+ <xref linkend="guc-vacuum-multixact-freeze-table-age"/> also controls
+ when <command>VACUUM</command> uses its aggressive strategy. This is an
+ independent Multixact ID based trigger for aggressive
+ <command>VACUUM</command>, which works just like
+ <varname>vacuum_freeze_table_age</varname>. It is applied against
+ <literal>mxid_age(relminmxid)</literal> at the start of each
+ <command>VACUUM</command>.
+ </para>
+
+ <para>
+ It doesn't matter if it was <varname>vacuum_freeze_table_age</varname> or
+ <varname>vacuum_multixact_freeze_table_age</varname> that triggered
+ <command>VACUUM</command>'s decision to use its aggressive strategy.
+ <emphasis>Every</emphasis> aggressive <command>VACUUM</command> will
+ advance <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> by applying the same generic policy
+ that controls which pages are frozen.
+ </para>
+
+ <para>
+ The default <varname>vacuum_freeze_table_age</varname> and
+ <varname>vacuum_multixact_freeze_table_age</varname> settings are
+ relatively low values. The <varname>vacuum_freeze_table_age</varname>
+ and <varname>vacuum_freeze_min_age</varname> defaults are intended to
+ limit the system to using only about 10% of the available transaction ID
+ space at any one time. This leaves the system with a generous amount of
+ <quote>slack capacity</quote> that allows XID allocations to continue in
+ the event of unforeseen problems with autovacuum and/or the application.
+ There might only be a negligible benefit from higher settings that aim to
+ reduce the number of <command>VACUUM</command>s that use the aggressive
+ strategy, in any case. Some applications may even
+ <emphasis>benefit</emphasis> from tuning that makes autovacuum perform
+ aggressive <command>VACUUM</command>s more often. If individual
+ aggressive <command>VACUUM</command>s can perform significantly less
+ <quote>catch-up</quote> freezing as a result, overall transaction
+ processing throughput is likely to be more stable and predictable.
+ </para>
+ </sect3>
+
+ <sect3 id="vacuum-antiwraparound-autovacuums">
+ <title>Anti-Wraparound Autovacuums</title>
+
+ <para>
+ To ensure that every table has its
+ <structfield>relfrozenxid</structfield> (and
+ <structfield>relminmxid</structfield>) advanced at regular intervals,
+ even in the case of completely static tables, autovacuum runs against any
+ table when it attains an age considered too far in the past. These are
+ <firstterm>anti-wraparound autovacuums</firstterm>. In practice, all
+ anti-wraparound autovacuums will use <command>VACUUM</command>'s
+ aggressive strategy (if they didn't, it would defeat the whole purpose of
+ anti-wraparound autovacuuming).
+ </para>
+
+ <para>
+ <xref linkend="guc-autovacuum-freeze-max-age"/> controls when the
+ autovacuum daemon launches anti-wraparound autovacuums. If the
+ <literal>age(relfrozenxid)</literal> of a table exceeds
+ <varname>autovacuum_freeze_max_age</varname> when the autovacuum daemon
+ periodically examines the database (which happens once every <xref
+ linkend="guc-autovacuum-naptime"/> seconds), then an anti-wraparound
+ autovacuum is launched against the table.
+ </para>
+
+ <para>
+ <xref linkend="guc-autovacuum-multixact-freeze-max-age"/> also controls
+ when the autovacuum daemon launches anti-wraparound autovacuums. It is
+ an independent Multixact ID based trigger for anti-wraparound
+ autovacuuming. If the <literal>mxid_age(relminmxid)</literal> of a table
+ exceeds <varname>autovacuum_multixact_freeze_max_age</varname> when the
+ autovacuum daemon periodically examines the database
+ <footnote>
+ <para>
+ Autovacuum may use a lower <quote>effective</quote> Multixact ID age
+ than the <varname>autovacuum_multixact_freeze_max_age</varname> setting
+ in <filename>postgresql.conf</filename>, though. Applying a lower
+ <quote>effective</quote> value like this avoids allowing the
+ <filename>pg_multixact/members</filename> <acronym>SLRU</acronym>
+ storage area to continue to grow in size for long, once its size
+ reaches <literal>2GB</literal>.
+ See <xref linkend="vacuum-truncate-xact-status"/>.
+ </para>
+ </footnote>, then an anti-wraparound autovacuum is launched against the
+ table.
+ </para>
+
+ <para>
+ Use of <command>VACUUM</command>'s aggressive strategy during
+ anti-wraparound autovacuuming is certain, because the effective value of
+ <varname>vacuum_freeze_table_age</varname> is silently limited to an
+ effective value no greater than 95% of the current value of
+ <varname>autovacuum_freeze_max_age</varname>. Similarly, the effective
+ value of <varname>vacuum_multixact_freeze_table_age</varname> is silently
+ limited to a value no greater than 95% of the current value of
+ <varname>autovacuum_multixact_freeze_max_age</varname>.
+ </para>
+
+ <para>
+ It doesn't matter if it was <varname>autovacuum_freeze_max_age</varname>
+ or <varname>autovacuum_multixact_freeze_max_age</varname> that triggered
+ an anti-wraparound autovacuum. <emphasis>Every</emphasis>
+ anti-wraparound autovacuum will be an aggressive
+ <command>VACUUM</command>, and will therefore advance
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> by applying the same generic policy
+ that controls which pages are frozen.
+ </para>
+
+ <para>
+ Anti-wraparound autovacuums are intended for static (and mostly static)
+ tables. There is no reason to expect that a table receiving continual
+ row inserts and/or row modifications will ever require an anti-wraparound
+ autovacuum. As a rule of thumb,
+ <varname>autovacuum_freeze_max_age</varname> should be set to a value
+ somewhat above <varname>vacuum_freeze_table_age</varname>, so that there
+ is a long window during which any autovacuum triggered by inserts,
+ updates, or deletes (or any manually issued <command>VACUUM</command>)
+ will become an aggressive <command>VACUUM</command>. This has the
+ advantage of allowing aggressive vacuuming to take place at a time when
+ vacuuming was required anyway. Each aggressive <command>VACUUM</command>
+ can therefore be expected to perform just as much useful work on
+ recovering disk space as an equivalent non-aggressive
+ <command>VACUUM</command> would have (had the non-aggressive strategy
+ been chosen instead).
+ </para>
+
+ <note>
+ <title>Aggressive/anti-wraparound differences</title>
+ <para>
+ Aggressive <command>VACUUM</command> is a special type of
+ <command>VACUUM</command>. It must advance
+ <structfield>relfrozenxid</structfield> up to a value that was no
+ greater than <varname>vacuum_freeze_min_age</varname> in age as of the
+ <emphasis>start</emphasis> of the <command>VACUUM</command> operation.
+ </para>
+ <para>
+ Anti-wraparound autovacuum is a special type of autovacuum. Its purpose
+ is to ensure that <structfield>relfrozenxid</structfield> advances when
+ no earlier <command>VACUUM</command> could advance it in passing —
+ often because no <command>VACUUM</command> has run against the table for
+ an extended period.
+ </para>
+ <para>
+ There is only one runtime behavioral difference between anti-wraparound
+ autovacuums and other autovacuums that run aggressive
+ <command>VACUUM</command>s: anti-wraparound autovacuums <emphasis>cannot
+ be autocancelled</emphasis>. This means that autovacuum workers that
+ perform anti-wraparound autovacuuming do not yield to conflicting
+ relation-level lock requests (e.g., from <command>ALTER
+ TABLE</command>). See <xref linkend="autovacuum-lock-conflicts"/> for
+ a full explanation.
+ </para>
+ </note>
+
+ <para>
+ In practice, anti-wraparound autovacuum is very likely to be the type of
+ autovacuum that updates the oldest <structfield>relfrozenxid</structfield>
+ in each database to a more recent value due to the presence of completely
+ static tables <footnote>
+ <para>
+ Anti-wraparound autovacuum is all but guaranteed to advance the oldest
+ <structfield>relfrozenxid</structfield>/<structfield>relminmxid</structfield>
+ (and therefore to advance
+ <structfield>datfrozenxid</structfield>/<structfield>datminmxid</structfield>)
+ in practice because in practice there is all but guaranteed to be at
+ least one totally static table that never gets an aggressive
+ <command>VACUUM</command> for any other reason (often just a tiny,
+ completely static system catalog table).
+ </para>
+ </footnote>. As discussed in <xref linkend="vacuum-aggressive"/>,
+ <structfield>datfrozenxid</structfield> only advances when the oldest
+ <structfield>relfrozenxid</structfield> in the database advances
+ (<structfield>relminmxid</structfield> likewise only advances when the
+ earliest <structfield>datminmxid</structfield> in the database advances).
+ This implies that anti-wraparound autovacuum is <emphasis>also</emphasis>
+ very likely to be involved when any database's
+ <structfield>datfrozenxid</structfield>/<structfield>datminmxid</structfield>
+ advances (and when the cluster-wide earliest unfrozen transaction
+ ID/Multixact ID is advanced to a more recent value, in turn).
+ </para>
+
+ <para>
+ It follows that <varname>autovacuum_freeze_max_age</varname> is usually
+ the limiting factor for advancing the cluster-wide oldest unfrozen
+ transaction ID found in <link linkend="functions-info-controldata"><filename>pg_control</filename></link>
+ (the cluster-wide oldest unfrozen Multixact ID might occasionally be
+ influenced by <varname>autovacuum_multixact_freeze_max_age</varname>,
+ too). This usually isn't much of a concern in itself, since it generally
+ doesn't predict anything about how far behind autovacuum is with freezing
+ physical heap pages. Note, however, that this effect
+ <emphasis>can</emphasis> significantly impact the amount of space
+ required to store transaction status information. The oldest transaction
+ status information (which is stored in external structures such as
+ <filename>pg_xact</filename>) cannot safely be truncated until
+ <command>VACUUM</command> can ascertain that there are no references to
+ the oldest entries remaining in any table, from any database. See <xref
+ linkend="vacuum-truncate-xact-status"/> for further details.
+ </para>
+ </sect3>
+
+ <sect3 id="vacuum-xid-exhaustion">
+ <title>Transaction ID exhaustion</title>
+ <para>
+ If for some reason autovacuum fails to advance any table's
+ <structfield>relfrozenxid</structfield> for an extended period (during
+ which transaction IDs continue to be allocated), the system will begin to
+ emit warning messages once the database's oldest XIDs attain an age
+ within forty million transactions of the 2.1 billion XID hard limit
+ described in <xref linkend="vacuum-aggressive"/>. For example:
<programlisting>
WARNING: database "mydb" must be vacuumed within 39985967 transactions
HINT: To avoid a database shutdown, execute a database-wide VACUUM in that database.
</programlisting>
- (A manual <command>VACUUM</command> should fix the problem, as suggested by the
- hint; but note that the <command>VACUUM</command> must be performed by a
- superuser, else it will fail to process system catalogs and thus not
- be able to advance the database's <structfield>datfrozenxid</structfield>.)
- If these warnings are
- ignored, the system will shut down and refuse to start any new
- transactions once there are fewer than three million transactions left
- until wraparound:
+ (A manual <command>VACUUM</command> should fix the problem, as suggested by the
+ hint; but note that the <command>VACUUM</command> must be performed by a
+ superuser, else it will fail to process system catalogs and thus not
+ be able to advance the database's <structfield>datfrozenxid</structfield>.)
+ If these warnings are ignored, the system will eventually refuse
+ to allocate new transaction IDs. This happens at the point that
+ there are fewer than three million transactions left:
<programlisting>
ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb"
HINT: Stop the postmaster and vacuum that database in single-user mode.
</programlisting>
- The three-million-transaction safety margin exists to let the
- administrator recover without data loss, by manually executing the
- required <command>VACUUM</command> commands. However, since the system will not
- execute commands once it has gone into the safety shutdown mode,
- the only way to do this is to stop the server and start the server in single-user
- mode to execute <command>VACUUM</command>. The shutdown mode is not enforced
- in single-user mode. See the <xref linkend="app-postgres"/> reference
- page for details about using single-user mode.
- </para>
-
- <sect3 id="vacuum-for-multixact-wraparound">
- <title>Multixacts and Wraparound</title>
-
- <indexterm>
- <primary>MultiXactId</primary>
- </indexterm>
-
- <indexterm>
- <primary>wraparound</primary>
- <secondary>of multixact IDs</secondary>
- </indexterm>
-
- <para>
- <firstterm>Multixact IDs</firstterm> are used to support row locking by
- multiple transactions. Since there is only limited space in a tuple
- header to store lock information, that information is encoded as
- a <quote>multiple transaction ID</quote>, or multixact ID for short,
- whenever there is more than one transaction concurrently locking a
- row. Information about which transaction IDs are included in any
- particular multixact ID is stored separately in
- the <filename>pg_multixact</filename> subdirectory, and only the multixact ID
- appears in the <structfield>xmax</structfield> field in the tuple header.
- Like transaction IDs, multixact IDs are implemented as a
- 32-bit counter and corresponding storage, all of which requires
- careful aging management, storage cleanup, and wraparound handling.
- There is a separate storage area which holds the list of members in
- each multixact, which also uses a 32-bit counter and which must also
- be managed.
+ The three-million-transaction safety margin exists to let the
+ administrator recover without data loss, by manually executing the
+ required <command>VACUUM</command> commands. However, since the system will not
+ execute commands once it has gone into the safety shutdown mode,
+ the only way to do this is to stop the server and start the server in single-user
+ mode to execute <command>VACUUM</command>. The shutdown mode is not enforced
+ in single-user mode. See the <xref linkend="app-postgres"/> reference
+ page for details about using single-user mode.
</para>
<para>
- Whenever <command>VACUUM</command> scans any part of a table, it will replace
- any multixact ID it encounters which is older than
- <xref linkend="guc-vacuum-multixact-freeze-min-age"/>
- by a different value, which can be the zero value, a single
- transaction ID, or a newer multixact ID. For each table,
- <structname>pg_class</structname>.<structfield>relminmxid</structfield> stores the oldest
- possible multixact ID still appearing in any tuple of that table.
- If this value is older than
- <xref linkend="guc-vacuum-multixact-freeze-table-age"/>, an aggressive
- vacuum is forced. As discussed in the previous section, an aggressive
- vacuum means that only those pages which are known to be all-frozen will
- be skipped. <function>mxid_age()</function> can be used on
- <structname>pg_class</structname>.<structfield>relminmxid</structfield> to find its age.
+ A similar safety mechanism is used to prevent Multixact ID allocations
+ when any table's <structfield>relminmxid</structfield> is dangerously far
+ in the past: Multixact ID exhaustion. If the system isn't experiencing
+ transaction ID exhaustion, Multixact ID exhaustion can be fixed
+ non-invasively by running a manual <command>VACUUM</command> without
+ entering single-user mode (otherwise follow the procedure for transaction
+ ID exhaustion). See <xref linkend="monitoring-table-age"/> for details
+ on how to determine which table's <structfield>relminmxid</structfield>
+ is dangerously far in the past.
</para>
- <para>
- Aggressive <command>VACUUM</command>s, regardless of what causes
- them, are <emphasis>guaranteed</emphasis> to be able to advance
- the table's <structfield>relminmxid</structfield>.
- Eventually, as all tables in all databases are scanned and their
- oldest multixact values are advanced, on-disk storage for older
- multixacts can be removed.
- </para>
-
- <para>
- As a safety device, an aggressive vacuum scan will
- occur for any table whose multixact-age is greater than <xref
- linkend="guc-autovacuum-multixact-freeze-max-age"/>. Also, if the
- storage occupied by multixacts members exceeds 2GB, aggressive vacuum
- scans will occur more often for all tables, starting with those that
- have the oldest multixact-age. Both of these kinds of aggressive
- scans will occur even if autovacuum is nominally disabled.
- </para>
+ <note>
+ <para>
+ Autovacuum has two different mechanisms that are designed to avoid
+ transaction ID exhaustion. The first mechansim is anti-wraparound
+ autovacuuming. There is an second, independent mechanism, used when
+ <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield> have already consumed a
+ significant fraction of the total available transaction ID space: the
+ failsafe.
+ </para>
+ <para>
+ The failsafe is triggered by <command>VACUUM</command> when the table's
+ <structfield>relfrozenxid</structfield> attains an age of <xref
+ linkend="guc-vacuum-failsafe-age"/> XIDs, or when the table's
+ <structfield>relminmxid</structfield> attains an age of <xref
+ linkend="guc-vacuum-multixact-failsafe-age"/> Multixact IDs. This
+ happens dynamically, when the risk of eventual transaction ID (or
+ Multixact ID) exhaustion is deemed to outweigh the risks of not
+ proceeding as planned with ordinary vacuuming.
+ </para>
+ <para>
+ Once the failsafe triggers, <command>VACUUM</command> prioritizes
+ advancing <structfield>relfrozenxid</structfield> and/or
+ <structfield>relminmxid</structfield> to avoid transaction ID
+ exhaustion. Most notably, <command>VACUUM</command> bypasses any
+ remaining non-essential maintenance, such as index vacuuming.
+ </para>
+ </note>
</sect3>
</sect2>
@@ -784,9 +1239,19 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
Vacuum maintains a <link linkend="storage-vm">visibility map</link> for each
table to keep track of which pages contain only tuples that are known to be
visible to all active transactions (and all future transactions, until the
- page is again modified). This has two purposes. First, vacuum
- itself can skip such pages on the next run, since there is nothing to
- clean up.
+ page is again modified). A separate bit tracks whether all of the tuples
+ are frozen.
+ </para>
+
+ <para>
+ The visibility map serves two purposes.
+ </para>
+
+ <para>
+ First, <command>VACUUM</command> itself can skip such pages on the
+ next run, since there is nothing to clean up. Even <link
+ linkend="vacuum-aggressive">aggressive <command>VACUUM</command>s</link>
+ can skip pages that are both all-visible and all-frozen.
</para>
<para>
@@ -806,6 +1271,79 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect2>
+ <sect2 id="vacuum-truncate-xact-status">
+ <title>Truncating Transaction Status Information</title>
+
+ <para>
+ As discussed in <xref linkend="vacuum-aggressive"/>, aggressive
+ autovacuuming plays a critical role in maintaining the XID address space
+ for the system as a whole. A secondary goal of this whole process is to
+ enable eventual truncation of the oldest transaction status information in
+ the <glossterm linkend="glossary-db-cluster">database cluster</glossterm>
+ as a whole. This status information is stored in dedicated <link
+ linkend="monitoring-pg-stat-slru-view">simple least-recently-used</link>
+ (<acronym>SLRU</acronym>) caches backed by external storage (see <xref
+ linkend="storage-file-layout"/>). Truncation is only possible when
+ <command>VACUUM</command> can ascertain that there are no references to
+ the oldest entries remaining in any table, from any database, by taking
+ the earliest <structfield>datfrozenxid</structfield> and
+ <structfield>datminmxid</structfield> among all databases in the cluster.
+ This isn't a maintenance task that affects individual tables; it's a
+ maintenance task that affects the whole cluster.
+ </para>
+
+ <para>
+ The space required to store transaction status information is likely to be
+ a low priority for most database administrators. It may occasionally be
+ useful to limit the maximum storage overhead used for transaction status
+ information by making anti-wraparound autovacuums happen more frequently.
+ The frequency of system-wide anti-wraparound autovacuuming increases when
+ <varname>autovacuum_freeze_max_age</varname> and
+ <varname>autovacuum_multixact_freeze_max_age</varname> are decreased in
+ <filename>postgresql.conf</filename>. This approach is effective (at
+ limiting the storage required for transaction status information) because
+ the <emphasis>oldest</emphasis> <structfield>datfrozenxid</structfield>
+ and <structfield>datminmxid</structfield> in the cluster are very likely
+ to depend on the frequency of anti-wraparound autovacuuming of completely
+ static tables. See <xref linkend="vacuum-antiwraparound-autovacuums"/> for
+ further discussion of the role of anti-wraparound autovacuuming in
+ advancing the cluster-wide oldest unfrozen transaction ID.
+ </para>
+
+ <para>
+ There are two <acronym>SLRU</acronym> storage areas associated with
+ transaction IDs. First, there is <filename>pg_xact</filename>, which
+ stores commit/abort status information. Second, there is
+ <filename>pg_commit_ts</filename>, which stores transaction commit
+ timestamps (when <xref linkend="guc-track-commit-timestamp"/> is set to
+ <literal>on</literal>). The default
+ <varname>autovacuum_freeze_max_age</varname> setting of 200 million
+ transactions translates to about 50MB of <filename>pg_xact</filename>
+ storage, and about 2GB of <filename>pg_commit_ts</filename> storage when
+ <varname>track_commit_timestamp</varname> is enabled (it is set to
+ <literal>off</literal> by default, which totally avoids the need to store
+ anything in <filename>pg_commit_ts</filename>).
+ </para>
+
+ <para>
+ There are also two <acronym>SLRU</acronym> storage areas associated with
+ Multixact IDs: <filename>pg_multixact/members</filename>, and
+ <filename>pg_multixact/offsets</filename>. These are logically one
+ storage area, implemented as two distinct storage areas. There is no
+ simple formula to determine the storage overhead per Multixact ID, since
+ Multixact IDs have a variable number of member transaction IDs (this is
+ what necessitates using two different physical storage areas). Note,
+ however, that if <filename>pg_multixact/members</filename> exceeds 2GB,
+ the effective value of <varname>autovacuum_multixact_freeze_max_age</varname>
+ used by autovacuum (and <command>VACUUM</command>) will be lower. This
+ results in more frequent
+ <link linkend="vacuum-antiwraparound-autovacuums">anti-wraparound
+ autovacuums</link> (since that's the only approach that reliably limits
+ the size of these storage areas). It might also increase the frequency of
+ aggressive <command>VACUUM</command>s more generally.
+ </para>
+ </sect2>
+
<sect2 id="vacuum-for-statistics">
<title>Updating Planner Statistics</title>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 99f7f95c3..0888342ef 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1032,9 +1032,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<row>
<entry><literal>BufferPin</literal></entry>
<entry>The server process is waiting for exclusive access to
- a data buffer. Buffer pin waits can be protracted if
- another process holds an open cursor that last read data from the
- buffer in question. See <xref linkend="wait-event-bufferpin-table"/>.
+ a data buffer. Buffer pin waits by aggressive
+ <command>VACUUM</command> (see <xref linkend="vacuum-aggressive"/>) can
+ be protracted if another process holds an for details on how to open
+ cursor that last read data from the buffer in question.
+ See <xref linkend="wait-event-bufferpin-table"/>.
</entry>
</row>
<row>
@@ -6246,6 +6248,78 @@ FROM pg_stat_get_backend_idset() AS backendid;
</sect2>
</sect1>
+ <sect1 id="monitoring-table-age">
+ <title>Monitoring table age</title>
+
+ <para>
+ It is crucial that autovacuum is able to run <command>VACUUM</command> (or
+ that every table is manually vacuumed) at somewhat regular intervals. Even
+ completely static tables need to participate in management of the
+ transaction ID space by <command>VACUUM</command>, as explained in <xref
+ linkend="vacuum-freezing-xid-space"/>.
+ </para>
+
+ <para>
+ This section provides details about how to monitor for this. You might
+ want to use a third party monitoring and alterting tool for this. Many
+ have off-the-shelf queries similar to the reference query shown here.
+ </para>
+
+ <caution>
+ <para>
+ Temporary tables are not maintained by autovacuum (see <xref
+ linkend="autovacuum-limitations"/>), but nevertheless have the same
+ requirements for freezing and <structfield>relfrozenxid</structfield>
+ advancement as permanent tables.
+ </para>
+ <para>
+ This means application code that <quote>leaks</quote> temporary tables can
+ threaten the availability of the system; eventually, the system will enter
+ a mode that makes it temporarily unable to allocate new transaction IDs
+ (see <xref linkend="vacuum-xid-exhaustion"/>), since autovacuum's usual
+ strategies for preventing that from happening cannot be used.
+ </para>
+ </caution>
+
+ <para>
+ A convenient way to examine information about
+ <structfield>relfrozenxid</structfield> and
+ <structfield>relminmxid</structfield> is to execute queries such as:
+
+<programlisting>
+SELECT c.oid::regclass as table_name,
+greatest(age(c.relfrozenxid),
+ age(t.relfrozenxid)) as xid_age,
+mxid_age(c.relminmxid)
+FROM pg_class c
+LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
+WHERE c.relkind IN ('r', 'm');
+
+SELECT datname,
+age(datfrozenxid) as xid_age,
+mxid_age(datminmxid)
+FROM pg_database;
+</programlisting>
+
+ The <function>age</function> function returns the number of transactions
+ from <structfield>relfrozenxid</structfield> to the next unallocated
+ transaction ID. The <function>mxid_age</function> function returns the
+ number of Multixact IDs from <structfield>relminmxid</structfield> to the
+ next unallocated Multixact ID.
+ </para>
+
+ <para>
+ The system should always have significant XID allocation slack capacity.
+ Ideally, the greatest
+ <literal>age(relfrozenxid)</literal>/<literal>age(datfrozenxid)</literal>
+ in the system will never be more than a fraction of the 2.1 billion XID
+ hard limit described in <xref linkend="vacuum-aggressive"/>. The default
+ <varname>vacuum_freeze_table_age</varname> setting of 200 million
+ transactions implies that the system should never use significantly more
+ than about 10% of that hard limit.
+ </para>
+ </sect1>
+
<sect1 id="monitoring-locks">
<title>Viewing Locks</title>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 10ef699fa..307504acb 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1514,11 +1514,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
If true, the autovacuum daemon will perform automatic <command>VACUUM</command>
and/or <command>ANALYZE</command> operations on this table following the rules
discussed in <xref linkend="autovacuum"/>.
- If false, this table will not be autovacuumed, except to prevent
- transaction ID wraparound. See <xref linkend="vacuum-for-wraparound"/> for
- more about wraparound prevention.
- Note that the autovacuum daemon does not run at all (except to prevent
- transaction ID wraparound) if the <xref linkend="guc-autovacuum"/>
+ If false, this table will not be autovacuumed, except to run
+ anti-wraparound autovacuum (see <xref linkend="vacuum-antiwraparound-autovacuums"/>).
+ Note that the autovacuum daemon does not run at all (except to run
+ anti-wraparound autovacuums) if the <xref linkend="guc-autovacuum"/>
parameter is false; setting individual tables' storage parameters does
not override that. Therefore there is seldom much point in explicitly
setting this storage parameter to <literal>true</literal>, only
diff --git a/doc/src/sgml/ref/prepare_transaction.sgml b/doc/src/sgml/ref/prepare_transaction.sgml
index f4f6118ac..719e0b25a 100644
--- a/doc/src/sgml/ref/prepare_transaction.sgml
+++ b/doc/src/sgml/ref/prepare_transaction.sgml
@@ -126,13 +126,13 @@ PREPARE TRANSACTION <replaceable class="parameter">transaction_id</replaceable>
<para>
It is unwise to leave transactions in the prepared state for a long time.
This will interfere with the ability of <command>VACUUM</command> to reclaim
- storage, and in extreme cases could cause the database to shut down
- to prevent transaction ID wraparound (see <xref
- linkend="vacuum-for-wraparound"/>). Keep in mind also that the transaction
- continues to hold whatever locks it held. The intended usage of the
- feature is that a prepared transaction will normally be committed or
- rolled back as soon as an external transaction manager has verified that
- other databases are also prepared to commit.
+ storage, and in extreme cases could cause the database to refuse to
+ allocate new transaction IDs (see <xref linkend="vacuum-xid-exhaustion"/>).
+ Keep in mind also that the transaction continues to hold whatever locks it
+ held. The intended usage of the feature is that a prepared transaction
+ will normally be committed or rolled back as soon as an external
+ transaction manager has verified that other databases are also prepared to
+ commit.
</para>
<para>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index 57bc4c23e..f8c010123 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -123,7 +123,9 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
<term><literal>FREEZE</literal></term>
<listitem>
<para>
- Selects aggressive <quote>freezing</quote> of tuples.
+ Makes <quote>freezing</quote> <emphasis>maximally</emphasis>
+ aggressive, and forces <command>VACUUM</command> to use its
+ <link linkend="vacuum-aggressive">aggressive strategy</link>.
Specifying <literal>FREEZE</literal> is equivalent to performing
<command>VACUUM</command> with the
<xref linkend="guc-vacuum-freeze-min-age"/> and
@@ -218,11 +220,11 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
<emphasis>always</emphasis> skip index vacuuming, even when
there are many dead tuples in the table. This may be useful
when it is necessary to make <command>VACUUM</command> run as
- quickly as possible to avoid imminent transaction ID wraparound
- (see <xref linkend="vacuum-for-wraparound"/>). However, the
- wraparound failsafe mechanism controlled by <xref
+ quickly as possible to avoid imminent transaction ID exhaustion
+ (see <xref linkend="vacuum-xid-exhaustion"/>). However, the failsafe
+ mechanism controlled by <xref
linkend="guc-vacuum-failsafe-age"/> will generally trigger
- automatically to avoid transaction ID wraparound failure, and
+ automatically to avoid transaction ID exhaustion failure, and
should be preferred. If index cleanup is not performed
regularly, performance may suffer, because as the table is
modified indexes will accumulate dead tuples and the table
@@ -232,7 +234,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ <replaceable class="paramet
<para>
This option has no effect for tables that have no index and is
ignored if the <literal>FULL</literal> option is used. It also
- has no effect on the transaction ID wraparound failsafe
+ has no effect on the transaction ID exhaustion failsafe
mechanism. When triggered it will skip index vacuuming, even
when <literal>INDEX_CLEANUP</literal> is set to
<literal>ON</literal>.
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index da2393783..9621fe378 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -231,9 +231,11 @@ PostgreSQL documentation
<para>
Only execute the vacuum or analyze commands on tables with a multixact
ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
- This setting is useful for prioritizing tables to process to prevent
- multixact ID wraparound (see
- <xref linkend="vacuum-for-multixact-wraparound"/>).
+ This setting is useful for prioritizing tables to advance
+ <structname>pg_class</structname>.<structfield>relfrozenxid</structfield>
+ for via an aggressive <command>VACUUM</command>
+ (see <xref linkend="vacuum-aggressive"/> and
+ <xref linkend="monitoring-table-age"/>).
</para>
<para>
For the purposes of this option, the multixact ID age of a relation is
@@ -252,9 +254,12 @@ PostgreSQL documentation
<para>
Only execute the vacuum or analyze commands on tables with a
transaction ID age of at least
- <replaceable class="parameter">xid_age</replaceable>. This setting
- is useful for prioritizing tables to process to prevent transaction
- ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ <replaceable class="parameter">xid_age</replaceable>. This
+ setting is useful for prioritizing tables to advance
+ <structname>pg_class</structname>.<structfield>relminmxid</structfield>
+ for via an aggressive <command>VACUUM</command>
+ (see <xref linkend="vacuum-aggressive"/> and
+ <xref linkend="monitoring-table-age"/>).
</para>
<para>
For the purposes of this option, the transaction ID age of a relation
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 148fb1b49..6d7ddc685 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -664,7 +664,8 @@ This information can also be used
by <link linkend="indexes-index-only-scans"><firstterm>index-only
scans</firstterm></link> to answer queries using only the index tuple.
The second bit, if set, means that all tuples on the page have been frozen.
-That means that even an anti-wraparound vacuum need not revisit the page.
+That means that even an aggressive vacuum (see <xref linkend="vacuum-aggressive"/>)
+need not revisit the page.
</para>
<para>
diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml
index 8a1f9fd6f..3cba050f0 100644
--- a/doc/src/sgml/xact.sgml
+++ b/doc/src/sgml/xact.sgml
@@ -180,7 +180,7 @@
rows and can be inspected using the <xref linkend="pgrowlocks"/>
extension. Row-level read locks might also require the assignment
of multixact IDs (<literal>mxid</literal>; see <xref
- linkend="vacuum-for-multixact-wraparound"/>).
+ linkend="vacuum-freezing-xid-space"/>).
</para>
</sect1>
--
2.40.1
[application/octet-stream] v4-0004-Reorder-routine-vacuuming-sections.patch (16.7K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/7-v4-0004-Reorder-routine-vacuuming-sections.patch)
download | inline diff:
From de7e1aaead4a8f8b4a680b7489a35fedad8059d2 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 11:19:50 -0700
Subject: [PATCH v4 4/9] Reorder routine vacuuming sections.
This doesn't change any of the content itself. It is a mechanical
change. The new order talks about maintenance tasks that happen within
the scope of the VACUUM command first, and then talks about ANALYZE
last. Furthermore, we talk about each maintenance task that happens
within the scope of VACUUM in an order that matches physical processing
order within vacuumlazy.c. (If you assume that "space-recovery" mostly
deals with pruning and "for-wraparound" mostly deals with freezing).
Old order:
<sect2 id="vacuum-basics">
<sect2 id="vacuum-for-space-recovery">
<sect2 id="vacuum-for-statistics">
<sect2 id="vacuum-for-visibility-map">
<sect2 id="vacuum-for-wraparound">
New order:
<sect2 id="vacuum-basics">
<sect2 id="vacuum-for-space-recovery">
<sect2 id="vacuum-for-wraparound">
<sect2 id="vacuum-for-visibility-map">
<sect2 id="vacuum-for-statistics">
A later commit will make the content that now appears in "vacuum-basics"
appear as the "Routine Vacuuming" sect1's introductory paragraph.
That'll make it easier to move advice about when to use VACUUM FULL to
some other chapter (since it isn't intended for "routine" use at all).
The new order will be easier to work with in later commits that overhaul
both "space-recovery" and "for-wraparound". Pruning and freezing are
related conceptually (e.g., holding back "removable cutoff"/OldestXmin
disrupts both in about the same way), which will be easier to discuss
with this ground work in place.
---
doc/src/sgml/maintenance.sgml | 300 +++++++++++++++++-----------------
1 file changed, 150 insertions(+), 150 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 702e2797c..83fa7ba8b 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -280,8 +280,9 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To update data statistics used by the
- <productname>PostgreSQL</productname> query planner.</simpara>
+ <simpara>To protect against loss of very old data due to
+ <firstterm>transaction ID wraparound</firstterm> or
+ <firstterm>multixact ID wraparound</firstterm>.</simpara>
</listitem>
<listitem>
@@ -291,9 +292,8 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
<listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ <simpara>To update data statistics used by the
+ <productname>PostgreSQL</productname> query planner.</simpara>
</listitem>
</orderedlist>
@@ -438,151 +438,6 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</tip>
</sect2>
- <sect2 id="vacuum-for-statistics">
- <title>Updating Planner Statistics</title>
-
- <indexterm zone="vacuum-for-statistics">
- <primary>statistics</primary>
- <secondary>of the planner</secondary>
- </indexterm>
-
- <indexterm zone="vacuum-for-statistics">
- <primary>ANALYZE</primary>
- </indexterm>
-
- <para>
- The <productname>PostgreSQL</productname> query planner relies on
- statistical information about the contents of tables in order to
- generate good plans for queries. These statistics are gathered by
- the <link linkend="sql-analyze"><command>ANALYZE</command></link> command,
- which can be invoked by itself or
- as an optional step in <command>VACUUM</command>. It is important to have
- reasonably accurate statistics, otherwise poor choices of plans might
- degrade database performance.
- </para>
-
- <para>
- The autovacuum daemon, if enabled, will automatically issue
- <command>ANALYZE</command> commands whenever the content of a table has
- changed sufficiently. However, administrators might prefer to rely
- on manually-scheduled <command>ANALYZE</command> operations, particularly
- if it is known that update activity on a table will not affect the
- statistics of <quote>interesting</quote> columns. The daemon schedules
- <command>ANALYZE</command> strictly as a function of the number of rows
- inserted or updated; it has no knowledge of whether that will lead
- to meaningful statistical changes.
- </para>
-
- <para>
- Tuples changed in partitions and inheritance children do not trigger
- analyze on the parent table. If the parent table is empty or rarely
- changed, it may never be processed by autovacuum, and the statistics for
- the inheritance tree as a whole won't be collected. It is necessary to
- run <command>ANALYZE</command> on the parent table manually in order to
- keep the statistics up to date.
- </para>
-
- <para>
- As with vacuuming for space recovery, frequent updates of statistics
- are more useful for heavily-updated tables than for seldom-updated
- ones. But even for a heavily-updated table, there might be no need for
- statistics updates if the statistical distribution of the data is
- not changing much. A simple rule of thumb is to think about how much
- the minimum and maximum values of the columns in the table change.
- For example, a <type>timestamp</type> column that contains the time
- of row update will have a constantly-increasing maximum value as
- rows are added and updated; such a column will probably need more
- frequent statistics updates than, say, a column containing URLs for
- pages accessed on a website. The URL column might receive changes just
- as often, but the statistical distribution of its values probably
- changes relatively slowly.
- </para>
-
- <para>
- It is possible to run <command>ANALYZE</command> on specific tables and even
- just specific columns of a table, so the flexibility exists to update some
- statistics more frequently than others if your application requires it.
- In practice, however, it is usually best to just analyze the entire
- database, because it is a fast operation. <command>ANALYZE</command> uses a
- statistically random sampling of the rows of a table rather than reading
- every single row.
- </para>
-
- <tip>
- <para>
- Although per-column tweaking of <command>ANALYZE</command> frequency might not be
- very productive, you might find it worthwhile to do per-column
- adjustment of the level of detail of the statistics collected by
- <command>ANALYZE</command>. Columns that are heavily used in <literal>WHERE</literal>
- clauses and have highly irregular data distributions might require a
- finer-grain data histogram than other columns. See <command>ALTER TABLE
- SET STATISTICS</command>, or change the database-wide default using the <xref
- linkend="guc-default-statistics-target"/> configuration parameter.
- </para>
-
- <para>
- Also, by default there is limited information available about
- the selectivity of functions. However, if you create a statistics
- object or an expression
- index that uses a function call, useful statistics will be
- gathered about the function, which can greatly improve query
- plans that use the expression index.
- </para>
- </tip>
-
- <tip>
- <para>
- The autovacuum daemon does not issue <command>ANALYZE</command> commands for
- foreign tables, since it has no means of determining how often that
- might be useful. If your queries require statistics on foreign tables
- for proper planning, it's a good idea to run manually-managed
- <command>ANALYZE</command> commands on those tables on a suitable schedule.
- </para>
- </tip>
-
- <tip>
- <para>
- The autovacuum daemon does not issue <command>ANALYZE</command> commands
- for partitioned tables. Inheritance parents will only be analyzed if the
- parent itself is changed - changes to child tables do not trigger
- autoanalyze on the parent table. If your queries require statistics on
- parent tables for proper planning, it is necessary to periodically run
- a manual <command>ANALYZE</command> on those tables to keep the statistics
- up to date.
- </para>
- </tip>
-
- </sect2>
-
- <sect2 id="vacuum-for-visibility-map">
- <title>Updating the Visibility Map</title>
-
- <para>
- Vacuum maintains a <link linkend="storage-vm">visibility map</link> for each
- table to keep track of which pages contain only tuples that are known to be
- visible to all active transactions (and all future transactions, until the
- page is again modified). This has two purposes. First, vacuum
- itself can skip such pages on the next run, since there is nothing to
- clean up.
- </para>
-
- <para>
- Second, it allows <productname>PostgreSQL</productname> to answer some
- queries using only the index, without reference to the underlying table.
- Since <productname>PostgreSQL</productname> indexes don't contain tuple
- visibility information, a normal index scan fetches the heap tuple for each
- matching index entry, to check whether it should be seen by the current
- transaction.
- An <link linkend="indexes-index-only-scans"><firstterm>index-only
- scan</firstterm></link>, on the other hand, checks the visibility map first.
- If it's known that all tuples on the page are
- visible, the heap fetch can be skipped. This is most useful on
- large data sets where the visibility map can prevent disk accesses.
- The visibility map is vastly smaller than the heap, so it can easily be
- cached even when the heap is very large.
- </para>
- </sect2>
-
<sect2 id="vacuum-for-wraparound">
<title>Preventing Transaction ID Wraparound Failures</title>
@@ -932,6 +787,151 @@ HINT: Stop the postmaster and vacuum that database in single-user mode.
</para>
</sect3>
</sect2>
+
+ <sect2 id="vacuum-for-visibility-map">
+ <title>Updating the Visibility Map</title>
+
+ <para>
+ Vacuum maintains a <link linkend="storage-vm">visibility map</link> for each
+ table to keep track of which pages contain only tuples that are known to be
+ visible to all active transactions (and all future transactions, until the
+ page is again modified). This has two purposes. First, vacuum
+ itself can skip such pages on the next run, since there is nothing to
+ clean up.
+ </para>
+
+ <para>
+ Second, it allows <productname>PostgreSQL</productname> to answer some
+ queries using only the index, without reference to the underlying table.
+ Since <productname>PostgreSQL</productname> indexes don't contain tuple
+ visibility information, a normal index scan fetches the heap tuple for each
+ matching index entry, to check whether it should be seen by the current
+ transaction.
+ An <link linkend="indexes-index-only-scans"><firstterm>index-only
+ scan</firstterm></link>, on the other hand, checks the visibility map first.
+ If it's known that all tuples on the page are
+ visible, the heap fetch can be skipped. This is most useful on
+ large data sets where the visibility map can prevent disk accesses.
+ The visibility map is vastly smaller than the heap, so it can easily be
+ cached even when the heap is very large.
+ </para>
+ </sect2>
+
+ <sect2 id="vacuum-for-statistics">
+ <title>Updating Planner Statistics</title>
+
+ <indexterm zone="vacuum-for-statistics">
+ <primary>statistics</primary>
+ <secondary>of the planner</secondary>
+ </indexterm>
+
+ <indexterm zone="vacuum-for-statistics">
+ <primary>ANALYZE</primary>
+ </indexterm>
+
+ <para>
+ The <productname>PostgreSQL</productname> query planner relies on
+ statistical information about the contents of tables in order to
+ generate good plans for queries. These statistics are gathered by
+ the <link linkend="sql-analyze"><command>ANALYZE</command></link> command,
+ which can be invoked by itself or
+ as an optional step in <command>VACUUM</command>. It is important to have
+ reasonably accurate statistics, otherwise poor choices of plans might
+ degrade database performance.
+ </para>
+
+ <para>
+ The autovacuum daemon, if enabled, will automatically issue
+ <command>ANALYZE</command> commands whenever the content of a table has
+ changed sufficiently. However, administrators might prefer to rely
+ on manually-scheduled <command>ANALYZE</command> operations, particularly
+ if it is known that update activity on a table will not affect the
+ statistics of <quote>interesting</quote> columns. The daemon schedules
+ <command>ANALYZE</command> strictly as a function of the number of rows
+ inserted or updated; it has no knowledge of whether that will lead
+ to meaningful statistical changes.
+ </para>
+
+ <para>
+ Tuples changed in partitions and inheritance children do not trigger
+ analyze on the parent table. If the parent table is empty or rarely
+ changed, it may never be processed by autovacuum, and the statistics for
+ the inheritance tree as a whole won't be collected. It is necessary to
+ run <command>ANALYZE</command> on the parent table manually in order to
+ keep the statistics up to date.
+ </para>
+
+ <para>
+ As with vacuuming for space recovery, frequent updates of statistics
+ are more useful for heavily-updated tables than for seldom-updated
+ ones. But even for a heavily-updated table, there might be no need for
+ statistics updates if the statistical distribution of the data is
+ not changing much. A simple rule of thumb is to think about how much
+ the minimum and maximum values of the columns in the table change.
+ For example, a <type>timestamp</type> column that contains the time
+ of row update will have a constantly-increasing maximum value as
+ rows are added and updated; such a column will probably need more
+ frequent statistics updates than, say, a column containing URLs for
+ pages accessed on a website. The URL column might receive changes just
+ as often, but the statistical distribution of its values probably
+ changes relatively slowly.
+ </para>
+
+ <para>
+ It is possible to run <command>ANALYZE</command> on specific tables and even
+ just specific columns of a table, so the flexibility exists to update some
+ statistics more frequently than others if your application requires it.
+ In practice, however, it is usually best to just analyze the entire
+ database, because it is a fast operation. <command>ANALYZE</command> uses a
+ statistically random sampling of the rows of a table rather than reading
+ every single row.
+ </para>
+
+ <tip>
+ <para>
+ Although per-column tweaking of <command>ANALYZE</command> frequency might not be
+ very productive, you might find it worthwhile to do per-column
+ adjustment of the level of detail of the statistics collected by
+ <command>ANALYZE</command>. Columns that are heavily used in <literal>WHERE</literal>
+ clauses and have highly irregular data distributions might require a
+ finer-grain data histogram than other columns. See <command>ALTER TABLE
+ SET STATISTICS</command>, or change the database-wide default using the <xref
+ linkend="guc-default-statistics-target"/> configuration parameter.
+ </para>
+
+ <para>
+ Also, by default there is limited information available about
+ the selectivity of functions. However, if you create a statistics
+ object or an expression
+ index that uses a function call, useful statistics will be
+ gathered about the function, which can greatly improve query
+ plans that use the expression index.
+ </para>
+ </tip>
+
+ <tip>
+ <para>
+ The autovacuum daemon does not issue <command>ANALYZE</command> commands for
+ foreign tables, since it has no means of determining how often that
+ might be useful. If your queries require statistics on foreign tables
+ for proper planning, it's a good idea to run manually-managed
+ <command>ANALYZE</command> commands on those tables on a suitable schedule.
+ </para>
+ </tip>
+
+ <tip>
+ <para>
+ The autovacuum daemon does not issue <command>ANALYZE</command> commands
+ for partitioned tables. Inheritance parents will only be analyzed if the
+ parent itself is changed - changes to child tables do not trigger
+ autoanalyze on the parent table. If your queries require statistics on
+ parent tables for proper planning, it is necessary to periodically run
+ a manual <command>ANALYZE</command> on those tables to keep the statistics
+ up to date.
+ </para>
+ </tip>
+
+ </sect2>
</sect1>
--
2.40.1
[application/octet-stream] v4-0007-Make-Routine-Vacuuming-autovacuum-orientated.patch (8.9K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/8-v4-0007-Make-Routine-Vacuuming-autovacuum-orientated.patch)
download | inline diff:
From ba27796ecce418105003442e957e8739d3f44c23 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 15:20:13 -0700
Subject: [PATCH v4 7/9] Make "Routine Vacuuming" autovacuum-orientated.
Now that it's no longer in its own sect2, shorten the "Vacuuming basics"
content, and make it more autovacuum-orientated. This gives much less
prominence to VACUUM FULL, which has little place in a section about
autovacuum. We no longer define avoiding the need to run VACUUM FULL as
the purpose of vacuuming.
A later commit that overhauls "Recovering Disk Space" will add back a
passing mention of things like VACUUM FULL and TRUNCATE, but only as
something that might be relevant in extreme cases. (Use of these
commands is hopefully neither "Routine" nor "Basic" to most users).
Also add some introductory information about the audience and goals of
the "Routine Vacuuming" section of the docs.
---
doc/src/sgml/maintenance.sgml | 132 +++++++++++++++++++++-------------
1 file changed, 83 insertions(+), 49 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 3f5b83b14..db8c5724e 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -32,11 +32,12 @@
</para>
<para>
- The other main category of maintenance task is periodic <quote>vacuuming</quote>
- of the database. This activity is discussed in
- <xref linkend="routine-vacuuming"/>. Closely related to this is updating
- the statistics that will be used by the query planner, as discussed in
- <xref linkend="vacuum-for-statistics"/>.
+ The other main category of maintenance task is periodic
+ <quote><link linkend="routine-vacuuming">vacuuming</link></quote> of
+ the database by autovacuum. Configuring autovacuum scheduling is
+ discussed in <xref linkend="autovacuum"/>. Autovacuum also updates
+ the statistics that will be used by the query planner, as discussed
+ in <xref linkend="vacuum-for-statistics"/>.
</para>
<para>
@@ -243,7 +244,7 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</sect1>
<sect1 id="routine-vacuuming">
- <title>Routine Vacuuming</title>
+ <title>Autovacuum Maintenance Tasks</title>
<indexterm zone="routine-vacuuming">
<primary>vacuum</primary>
@@ -251,24 +252,18 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<para>
<productname>PostgreSQL</productname> databases require periodic
- maintenance known as <firstterm>vacuuming</firstterm>. For many installations, it
- is sufficient to let vacuuming be performed by the <firstterm>autovacuum
- daemon</firstterm>, which is described in <xref linkend="autovacuum"/>. You might
- need to adjust the autovacuuming parameters described there to obtain best
- results for your situation. Some database administrators will want to
- supplement or replace the daemon's activities with manually-managed
- <command>VACUUM</command> commands, which typically are executed according to a
- schedule by <application>cron</application> or <application>Task
- Scheduler</application> scripts. To set up manually-managed vacuuming properly,
- it is essential to understand the issues discussed in the next few
- subsections. Administrators who rely on autovacuuming may still wish
- to skim this material to help them understand and adjust autovacuuming.
+ maintenance known as <firstterm>vacuuming</firstterm>, and require
+ periodic updates to the statistics used by the
+ <productname>PostgreSQL</productname> query planner. The <link
+ linkend="sql-vacuum"><command>VACUUM</command></link> and <link
+ linkend="sql-analyze"><command>ANALYZE</command></link> commands
+ perform these maintenance tasks. The <firstterm>autovacuum
+ daemon</firstterm> automatically schedules maintenance tasks based on
+ workload requirements.
</para>
-
<para>
- <productname>PostgreSQL</productname>'s
- <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
- process each table on a regular basis for several reasons:
+ The autovacuum daemon has to process each table regularly for several
+ reasons:
<orderedlist>
<listitem>
@@ -294,35 +289,74 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
</listitem>
</orderedlist>
- Each of these reasons dictates performing <command>VACUUM</command> operations
- of varying frequency and scope, as explained in the following subsections.
+ The first four maintenance tasks are handled by running
+ <command>VACUUM</command> from within an autovacuum worker process. The
+ fifth and final task (maintenance of planner statistics) is handled by
+ running <command>ANALYZE</command> from within an autovacuum worker
+ process.
+ </para>
+ <para>
+ Generally speaking, database administrators new to tuning autovacuum should
+ start by considering the need to adjust autovacuum's scheduling.
+ Autovacuum scheduling is controlled via threshold settings. These settings
+ determine when autovacuum should launch a worker to run
+ <command>VACUUM</command> and/or <command>ANALYZE</command>; see the
+ previous section, <xref linkend="autovacuum"/>. This section provides
+ additional information about the design and goals of autovacuum,
+ <command>VACUUM</command>, and <command>ANALYZE</command>. The intended
+ audience is database administrators that wish to perform more advanced
+ autovacuum tuning, with any of the following goals in mind:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Tuning <command>VACUUM</command> to improve query response times.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Making sure that <command>VACUUM</command>'s management of the
+ transaction ID address space is functioning optimally.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Tuning <command>VACUUM</command> for performance stability.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
+ With larger installations, tuning autovacuum usually won't be a once-off
+ task; it is best to approach tuning as an iterative, applied process.
+ </para>
+ <para>
+ Autovacuum might create a lot of I/O traffic at times, which can cause poor
+ performance for other active sessions. There are configuration parameters
+ you can adjust to reduce the impact on system response time. See the
+ autovacuum-specific cost delay settings described in
+ <xref linkend="runtime-config-autovacuum"/>, and additional cost delay
+ settings described in <xref linkend="runtime-config-resource-vacuum-cost"/>.
+ </para>
+ <para>
+ Database administrators might also find it useful to supplement the
+ daemon's activities with manually-managed <command>VACUUM</command>
+ commands. Scripting tools like <application>cron</application> and
+ <application>Task Manager</application> can help with this. It can be
+ useful to perform off-hours <command>VACUUM</command> commands during
+ periods when the application experiences less demand (e.g., on weekends, or
+ in the middle of the night). This section applies equally to
+ manually-issued <command>VACUUM</command> and <command>ANALYZE</command>
+ operations, except where otherwise noted.
</para>
- <para>
- There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
- and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
- disk space but runs much more slowly. Also,
- the standard form of <command>VACUUM</command> can run in parallel with production
- database operations. (Commands such as <command>SELECT</command>,
- <command>INSERT</command>, <command>UPDATE</command>, and
- <command>DELETE</command> will continue to function normally, though you
- will not be able to modify the definition of a table with commands such as
- <command>ALTER TABLE</command> while it is being vacuumed.)
- <command>VACUUM FULL</command> requires an
- <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
- working on, and therefore cannot be done in parallel with other use
- of the table. Generally, therefore,
- administrators should strive to use standard <command>VACUUM</command> and
- avoid <command>VACUUM FULL</command>.
- </para>
-
- <para>
- <command>VACUUM</command> creates a substantial amount of I/O
- traffic, which can cause poor performance for other active sessions.
- There are configuration parameters that can be adjusted to reduce the
- performance impact of background vacuuming — see
- <xref linkend="runtime-config-resource-vacuum-cost"/>.
- </para>
+ <tip>
+ <para>
+ You can monitor <command>VACUUM</command> progress (whether run by
+ autovacuum or manually) via the
+ <structname>pg_stat_progress_vacuum</structname> view. See
+ <xref linkend="vacuum-progress-reporting"/>.
+ </para>
+ </tip>
<sect2 id="vacuum-for-space-recovery">
<title>Recovering Disk Space</title>
--
2.40.1
[application/octet-stream] v4-0006-Merge-basic-vacuuming-sect2-into-sect1-introducti.patch (6.1K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/9-v4-0006-Merge-basic-vacuuming-sect2-into-sect1-introducti.patch)
download | inline diff:
From 8766671e9de5fff2cc7fa39bfc64bf14b12b9674 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 11:44:45 -0700
Subject: [PATCH v4 6/9] Merge "basic vacuuming" sect2 into sect1 introduction.
This doesn't change any of the content itself. It just merges the
original text into the sect1 text that immediately preceded it.
This is preparation for the next commit, which will remove most of the
text "relocated" in this commit. This structure should make things a
little easier for doc translators.
---
doc/src/sgml/maintenance.sgml | 106 ++++++++++++++++------------------
1 file changed, 51 insertions(+), 55 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 970c4a848..3f5b83b14 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -265,68 +265,64 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
to skim this material to help them understand and adjust autovacuuming.
</para>
- <sect2 id="vacuum-basics">
- <title>Vacuuming Basics</title>
+ <para>
+ <productname>PostgreSQL</productname>'s
+ <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
+ process each table on a regular basis for several reasons:
- <para>
- <productname>PostgreSQL</productname>'s
- <link linkend="sql-vacuum"><command>VACUUM</command></link> command has to
- process each table on a regular basis for several reasons:
+ <orderedlist>
+ <listitem>
+ <simpara>To recover or reuse disk space occupied by updated or deleted
+ rows.</simpara>
+ </listitem>
- <orderedlist>
- <listitem>
- <simpara>To recover or reuse disk space occupied by updated or deleted
- rows.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To protect against loss of very old data due to
+ <firstterm>transaction ID wraparound</firstterm> or
+ <firstterm>multixact ID wraparound</firstterm>.</simpara>
+ </listitem>
- <listitem>
- <simpara>To protect against loss of very old data due to
- <firstterm>transaction ID wraparound</firstterm> or
- <firstterm>multixact ID wraparound</firstterm>.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To update the visibility map, which speeds
+ up <link linkend="indexes-index-only-scans">index-only
+ scans</link>.</simpara>
+ </listitem>
- <listitem>
- <simpara>To update the visibility map, which speeds
- up <link linkend="indexes-index-only-scans">index-only
- scans</link>.</simpara>
- </listitem>
+ <listitem>
+ <simpara>To update data statistics used by the
+ <productname>PostgreSQL</productname> query planner.</simpara>
+ </listitem>
+ </orderedlist>
- <listitem>
- <simpara>To update data statistics used by the
- <productname>PostgreSQL</productname> query planner.</simpara>
- </listitem>
- </orderedlist>
+ Each of these reasons dictates performing <command>VACUUM</command> operations
+ of varying frequency and scope, as explained in the following subsections.
+ </para>
- Each of these reasons dictates performing <command>VACUUM</command> operations
- of varying frequency and scope, as explained in the following subsections.
- </para>
+ <para>
+ There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
+ and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
+ disk space but runs much more slowly. Also,
+ the standard form of <command>VACUUM</command> can run in parallel with production
+ database operations. (Commands such as <command>SELECT</command>,
+ <command>INSERT</command>, <command>UPDATE</command>, and
+ <command>DELETE</command> will continue to function normally, though you
+ will not be able to modify the definition of a table with commands such as
+ <command>ALTER TABLE</command> while it is being vacuumed.)
+ <command>VACUUM FULL</command> requires an
+ <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
+ working on, and therefore cannot be done in parallel with other use
+ of the table. Generally, therefore,
+ administrators should strive to use standard <command>VACUUM</command> and
+ avoid <command>VACUUM FULL</command>.
+ </para>
- <para>
- There are two variants of <command>VACUUM</command>: standard <command>VACUUM</command>
- and <command>VACUUM FULL</command>. <command>VACUUM FULL</command> can reclaim more
- disk space but runs much more slowly. Also,
- the standard form of <command>VACUUM</command> can run in parallel with production
- database operations. (Commands such as <command>SELECT</command>,
- <command>INSERT</command>, <command>UPDATE</command>, and
- <command>DELETE</command> will continue to function normally, though you
- will not be able to modify the definition of a table with commands such as
- <command>ALTER TABLE</command> while it is being vacuumed.)
- <command>VACUUM FULL</command> requires an
- <literal>ACCESS EXCLUSIVE</literal> lock on the table it is
- working on, and therefore cannot be done in parallel with other use
- of the table. Generally, therefore,
- administrators should strive to use standard <command>VACUUM</command> and
- avoid <command>VACUUM FULL</command>.
- </para>
-
- <para>
- <command>VACUUM</command> creates a substantial amount of I/O
- traffic, which can cause poor performance for other active sessions.
- There are configuration parameters that can be adjusted to reduce the
- performance impact of background vacuuming — see
- <xref linkend="runtime-config-resource-vacuum-cost"/>.
- </para>
- </sect2>
+ <para>
+ <command>VACUUM</command> creates a substantial amount of I/O
+ traffic, which can cause poor performance for other active sessions.
+ There are configuration parameters that can be adjusted to reduce the
+ performance impact of background vacuuming — see
+ <xref linkend="runtime-config-resource-vacuum-cost"/>.
+ </para>
<sect2 id="vacuum-for-space-recovery">
<title>Recovering Disk Space</title>
--
2.40.1
[application/octet-stream] v4-0008-Overhaul-Recovering-Disk-Space-vacuuming-docs.patch (11.4K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/10-v4-0008-Overhaul-Recovering-Disk-Space-vacuuming-docs.patch)
download | inline diff:
From 064cfa0b489a2c76dd8b527e119ca5c4658de295 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 12:33:42 -0700
Subject: [PATCH v4 8/9] Overhaul "Recovering Disk Space" vacuuming docs.
XXX This commit is much less worked out and polished than the work on
freezing. It should very much be considered a work in progress, and
isn't the priority for now.
Say a lot more about the possible impact of long-running transactions on
VACUUM. Remove all talk of administrators getting by without
autovacuum; at most administrators might want to schedule manual VACUUM
operations to supplement autovacuum (this documentation was written at a
time when the visibility map didn't exist, even in its most basic form).
Also describe VACUUM FULL as an entirely different kind of operation to
conventional lazy vacuum.
XXX Open question for this commit:
I wonder if it would make sense to move all of that stuff into its own
new sect1 of "Chapter 29. Monitoring Disk Usage" -- something along
the lines of "what to do about bloat when all else fails, when the
problem gets completely out of hand". Naturally we'd link to this new
section from "Routine Vacuuming".
XXX For now, a lot of the information about CLUSTER and VACUUM FULL is
moved into Note/Warning boxes. This arrangement is definitely going to
be temporary.
---
doc/src/sgml/maintenance.sgml | 165 ++++++++++++++++++----------------
1 file changed, 87 insertions(+), 78 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index db8c5724e..f00442564 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -372,100 +372,109 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
This approach is necessary to gain the benefits of multiversion
concurrency control (<acronym>MVCC</acronym>, see <xref linkend="mvcc"/>): the row version
must not be deleted while it is still potentially visible to other
- transactions. But eventually, an outdated or deleted row version is no
- longer of interest to any transaction. The space it occupies must then be
- reclaimed for reuse by new rows, to avoid unbounded growth of disk
- space requirements. This is done by running <command>VACUUM</command>.
+ transactions. A deleted row version (whether from an
+ <command>UPDATE</command> or <command>DELETE</command>) will usually cease
+ to be of interest to any still-running transaction shortly after the
+ original deleting transaction commits.
</para>
<para>
- The standard form of <command>VACUUM</command> removes dead row
- versions in tables and indexes and marks the space available for
- future reuse. However, it will not return the space to the operating
- system, except in the special case where one or more pages at the
- end of a table become entirely free and an exclusive table lock can be
- easily obtained. In contrast, <command>VACUUM FULL</command> actively compacts
- tables by writing a complete new version of the table file with no dead
- space. This minimizes the size of the table, but can take a long time.
- It also requires extra disk space for the new copy of the table, until
- the operation completes.
+ The space dead tuples occupy must eventually be reclaimed for reuse by new
+ rows, to avoid unbounded growth of disk space requirements. Reclaiming
+ space from dead rows is <command>VACUUM</command>'s main responsibility.
</para>
<para>
- The usual goal of routine vacuuming is to do standard <command>VACUUM</command>s
- often enough to avoid needing <command>VACUUM FULL</command>. The
- autovacuum daemon attempts to work this way, and in fact will
- never issue <command>VACUUM FULL</command>. In this approach, the idea
- is not to keep tables at their minimum size, but to maintain steady-state
- usage of disk space: each table occupies space equivalent to its
- minimum size plus however much space gets used up between vacuum runs.
- Although <command>VACUUM FULL</command> can be used to shrink a table back
- to its minimum size and return the disk space to the operating system,
- there is not much point in this if the table will just grow again in the
- future. Thus, moderately-frequent standard <command>VACUUM</command> runs are a
- better approach than infrequent <command>VACUUM FULL</command> runs for
- maintaining heavily-updated tables.
+ The <glossterm linkend="glossary-xid">transaction ID number
+ (<acronym>XID</acronym>)</glossterm> based cutoff point that
+ <command>VACUUM</command> uses to determine if a deleted tuple is safe to
+ physically remove is reported under <literal>removable cutoff</literal> in
+ the server log when autovacuum logging (controlled by <xref
+ linkend="guc-log-autovacuum-min-duration"/>) reports on a
+ <command>VACUUM</command> operation executed by autovacuum. Tuples that
+ are not yet safe to remove are counted as <literal>dead but not yet
+ removable</literal> tuples in the log report. <command>VACUUM</command>
+ establishes its <literal>removable cutoff</literal> once, at the start of
+ the operation. Any older <acronym>MVCC</acronym> snapshot (or transaction
+ that allocates an XID) that's still running when the cutoff is established
+ may hold it back.
</para>
- <para>
- Some administrators prefer to schedule vacuuming themselves, for example
- doing all the work at night when load is low.
- The difficulty with doing vacuuming according to a fixed schedule
- is that if a table has an unexpected spike in update activity, it may
- get bloated to the point that <command>VACUUM FULL</command> is really necessary
- to reclaim space. Using the autovacuum daemon alleviates this problem,
- since the daemon schedules vacuuming dynamically in response to update
- activity. It is unwise to disable the daemon completely unless you
- have an extremely predictable workload. One possible compromise is
- to set the daemon's parameters so that it will only react to unusually
- heavy update activity, thus keeping things from getting out of hand,
- while scheduled <command>VACUUM</command>s are expected to do the bulk of the
- work when the load is typical.
- </para>
+ <caution>
+ <para>
+ It's critical that no long-running transactions are allowed to hold back
+ every <command>VACUUM</command> operation's cutoff for an extended
+ period. It may be a good idea to add monitoring to alert you about this.
+ </para>
+ </caution>
+
+ <note>
+ <para>
+ <command>VACUUM</command> can remove tuples inserted by aborted
+ transactions immediately
+ </para>
+ </note>
<para>
- For those not using autovacuum, a typical approach is to schedule a
- database-wide <command>VACUUM</command> once a day during a low-usage period,
- supplemented by more frequent vacuuming of heavily-updated tables as
- necessary. (Some installations with extremely high update rates vacuum
- their busiest tables as often as once every few minutes.) If you have
- multiple databases in a cluster, don't forget to
- <command>VACUUM</command> each one; the program <xref
- linkend="app-vacuumdb"/> might be helpful.
+ <command>VACUUM</command> usually doesn't return space to the operating
+ system. There is one exception: space is returned to the OS whenever a
+ group of contiguous pages appears at the end of a table.
+ <command>VACUUM</command> must acquire an <literal>ACCESS
+ EXCLUSIVE</literal> lock to perform relation truncation. You can disable
+ relation truncation by setting the table's
+ <varname>vacuum_truncate</varname> storage parameter to
+ <literal>off</literal>.
</para>
<tip>
- <para>
- Plain <command>VACUUM</command> may not be satisfactory when
- a table contains large numbers of dead row versions as a result of
- massive update or delete activity. If you have such a table and
- you need to reclaim the excess disk space it occupies, you will need
- to use <command>VACUUM FULL</command>, or alternatively
- <link linkend="sql-cluster"><command>CLUSTER</command></link>
- or one of the table-rewriting variants of
- <link linkend="sql-altertable"><command>ALTER TABLE</command></link>.
- These commands rewrite an entire new copy of the table and build
- new indexes for it. All these options require an
- <literal>ACCESS EXCLUSIVE</literal> lock. Note that
- they also temporarily use extra disk space approximately equal to the size
- of the table, since the old copies of the table and indexes can't be
- released until the new ones are complete.
- </para>
+ <para>
+ If you have a table whose entire contents are deleted periodically,
+ consider using <command>TRUNCATE</command> rather than
+ <command>DELETE</command>. <command>TRUNCATE</command> removes the entire
+ table's contents immediately, obviating the need for
+ <command>VACUUM</command>. One disadvantage is that strict
+ <acronym>MVCC</acronym> semantics are violated.
+ </para>
</tip>
-
<tip>
- <para>
- If you have a table whose entire contents are deleted on a periodic
- basis, consider doing it with
- <link linkend="sql-truncate"><command>TRUNCATE</command></link> rather
- than using <command>DELETE</command> followed by
- <command>VACUUM</command>. <command>TRUNCATE</command> removes the
- entire content of the table immediately, without requiring a
- subsequent <command>VACUUM</command> or <command>VACUUM
- FULL</command> to reclaim the now-unused disk space.
- The disadvantage is that strict MVCC semantics are violated.
- </para>
+ <para>
+ <command>VACUUM FULL</command> (or <command>CLUSTER</command>) can be
+ useful when dealing with extreme amounts of dead tuples. It can reclaim
+ more disk space, but it is much slower, and usually more disruptive.
+ <command>VACUUM FULL</command> rewrites an entire new copy of the table
+ and rebuilds all of the table's indexes. This makes it suitable for
+ highly fragmented tables, and tables where significant amounts of space
+ can be reclaimed.
+ </para>
</tip>
+ <note>
+ <para>
+ Although <command>VACUUM FULL</command> is technically an option of the
+ <command>VACUUM</command> command, <command>VACUUM FULL</command> uses a
+ completely different implementation. <command>VACUUM FULL</command> is
+ essentially a variant of <command>CLUSTER</command>. (The name
+ <command>VACUUM FULL</command> is historical; the original implementation
+ was closer to standard <command>VACUUM</command>.)
+ </para>
+ </note>
+ <warning>
+ <para>
+ <command>TRUNCATE</command>, <command>VACUUM FULL</command>, and
+ <command>CLUSTER</command> all require an <literal>ACCESS
+ EXCLUSIVE</literal> lock, which can be highly disruptive
+ (<command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command>, and <command>DELETE</command> commands can't
+ run at the same time).
+ </para>
+ </warning>
+ <warning>
+ <para>
+ <command>VACUUM FULL</command> and <command>CLUSTER</command> temporarily
+ use extra disk space. The extra space required is approximately equal to
+ the size of the table, since the old copies of the table and indexes
+ can't be released until the new ones are complete.
+ </para>
+ </warning>
</sect2>
<sect2 id="vacuum-for-wraparound">
--
2.40.1
[application/octet-stream] v4-0005-Move-Interpreting-XID-stamps-from-tuple-headers.patch (11.6K, ../../CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1Fx_KmrdOVoPZXkpA-TFwg@mail.gmail.com/11-v4-0005-Move-Interpreting-XID-stamps-from-tuple-headers.patch)
download | inline diff:
From fbd1260730a7e9a22033b24db2a1a8e7c4d58ef7 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <[email protected]>
Date: Sat, 22 Apr 2023 12:41:00 -0700
Subject: [PATCH v4 5/9] Move Interpreting XID stamps from tuple headers.
Move handling of 32-bit XID comparisons/physical wraparound from
"Routine Vacuuming" to chapter about transaction internals.
This is intended to be fairly close to a mechanical change. It isn't
entirely mechanical, though, since the original wording has been
slightly modified for it to work in context.
TODO fix xact.sgml indentation. The new content is indented correctly
already, but the existing content will need to be re-indented to match
in a later commit. As always, structuring things this way is intended
to make life a little bit easier for doc translators.
---
doc/src/sgml/maintenance.sgml | 80 +++++++---------------------------
doc/src/sgml/xact.sgml | 82 ++++++++++++++++++++++++++++++++++-
2 files changed, 96 insertions(+), 66 deletions(-)
diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 83fa7ba8b..970c4a848 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -446,75 +446,25 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
<secondary>wraparound</secondary>
</indexterm>
- <indexterm>
- <primary>wraparound</primary>
- <secondary>of transaction IDs</secondary>
- </indexterm>
+ <indexterm>
+ <primary>wraparound</primary>
+ <secondary>of transaction IDs</secondary>
+ </indexterm>
<para>
- <productname>PostgreSQL</productname>'s
- <link linkend="mvcc-intro">MVCC</link> transaction semantics
- depend on being able to compare transaction ID (<acronym>XID</acronym>)
- numbers: a row version with an insertion XID greater than the current
- transaction's XID is <quote>in the future</quote> and should not be visible
- to the current transaction. But since transaction IDs have limited size
- (32 bits) a cluster that runs for a long time (more
- than 4 billion transactions) would suffer <firstterm>transaction ID
- wraparound</firstterm>: the XID counter wraps around to zero, and all of a sudden
- transactions that were in the past appear to be in the future — which
- means their output become invisible. In short, catastrophic data loss.
- (Actually the data is still there, but that's cold comfort if you cannot
- get at it.) To avoid this, it is necessary to vacuum every table
- in every database at least once every two billion transactions.
+ <productname>PostgreSQL</productname>'s <link
+ linkend="mvcc-intro">MVCC</link> transaction semantics depend on
+ being able to compare <glossterm linkend="glossary-xid">transaction
+ ID numbers (<acronym>XID</acronym>)</glossterm> to determine
+ whether or not the row is visible to each query's MVCC snapshot
+ (see <xref linkend="interpreting-xid-stamps"/>). But since
+ on-disk storage of transaction IDs in heap pages uses a truncated
+ 32-bit representation to save space (rather than the full 64-bit
+ representation), it is necessary to vacuum every table in every
+ database <emphasis>at least</emphasis> once every two billion
+ transactions (though far more frequent vacuuming is typical).
</para>
- <para>
- The reason that periodic vacuuming solves the problem is that
- <command>VACUUM</command> will mark rows as <emphasis>frozen</emphasis>, indicating that
- they were inserted by a transaction that committed sufficiently far in
- the past that the effects of the inserting transaction are certain to be
- visible to all current and future transactions.
- Normal XIDs are
- compared using modulo-2<superscript>32</superscript> arithmetic. This means
- that for every normal XID, there are two billion XIDs that are
- <quote>older</quote> and two billion that are <quote>newer</quote>; another
- way to say it is that the normal XID space is circular with no
- endpoint. Therefore, once a row version has been created with a particular
- normal XID, the row version will appear to be <quote>in the past</quote> for
- the next two billion transactions, no matter which normal XID we are
- talking about. If the row version still exists after more than two billion
- transactions, it will suddenly appear to be in the future. To
- prevent this, <productname>PostgreSQL</productname> reserves a special XID,
- <literal>FrozenTransactionId</literal>, which does not follow the normal XID
- comparison rules and is always considered older
- than every normal XID.
- Frozen row versions are treated as if the inserting XID were
- <literal>FrozenTransactionId</literal>, so that they will appear to be
- <quote>in the past</quote> to all normal transactions regardless of wraparound
- issues, and so such row versions will be valid until deleted, no matter
- how long that is.
- </para>
-
- <note>
- <para>
- In <productname>PostgreSQL</productname> versions before 9.4, freezing was
- implemented by actually replacing a row's insertion XID
- with <literal>FrozenTransactionId</literal>, which was visible in the
- row's <structname>xmin</structname> system column. Newer versions just set a flag
- bit, preserving the row's original <structname>xmin</structname> for possible
- forensic use. However, rows with <structname>xmin</structname> equal
- to <literal>FrozenTransactionId</literal> (2) may still be found
- in databases <application>pg_upgrade</application>'d from pre-9.4 versions.
- </para>
- <para>
- Also, system catalogs may contain rows with <structname>xmin</structname> equal
- to <literal>BootstrapTransactionId</literal> (1), indicating that they were
- inserted during the first phase of <application>initdb</application>.
- Like <literal>FrozenTransactionId</literal>, this special XID is treated as
- older than every normal XID.
- </para>
- </note>
-
<para>
<xref linkend="guc-vacuum-freeze-min-age"/>
controls how old an XID value has to be before rows bearing that XID will be
diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml
index b467660ee..8a1f9fd6f 100644
--- a/doc/src/sgml/xact.sgml
+++ b/doc/src/sgml/xact.sgml
@@ -22,6 +22,8 @@
single-statement transactions.
</para>
+ <sect2 id="virtual-xids">
+ <title>Virtual Transaction IDs</title>
<para>
Every transaction is identified by a unique
<literal>VirtualTransactionId</literal> (also called
@@ -46,10 +48,13 @@
started, particularly if the transaction started with statements that
only performed database reads.
</para>
+ </sect2>
+ <sect2 id="permanent-xids">
+ <title>Permanent Transaction IDs</title>
<para>
The internal transaction ID type <type>xid</type> is 32 bits wide
- and <link linkend="vacuum-for-wraparound">wraps around</link> every
+ and wraps around every
4 billion transactions. A 32-bit epoch is incremented during each
wraparound. There is also a 64-bit type <type>xid8</type> which
includes this epoch and therefore does not wrap around during the
@@ -69,6 +74,80 @@
linkend="guc-track-commit-timestamp"/> is enabled.
</para>
+ <sect3 id="interpreting-xid-stamps">
+ <title><type>TransactionId</type> comparison rules</title>
+ <para>
+ The system often needs to compare <structfield>t_xmin</structfield>
+ and <structfield>t_xmax</structfield> fields for MVCC snapshot
+ visibility checks.
+ </para>
+
+ <para>
+ We use a truncated 32-bit representation of transaction IDs, rather than
+ using the full 64-bit representation. The 2.1 billion XIDs
+ <quote>distance</quote> invariant must be preserved because transaction
+ IDs stored in heap row headers use a truncated 32-bit representation
+ (rather than the full 64-bit representation). Since all unfrozen
+ transaction IDs from heap tuple headers <emphasis>must</emphasis> be from
+ the same transaction ID epoch (or from a space in the 64-bit
+ representation that spans two adjoining transaction ID epochs), there
+ isn't any need to store a separate epoch field in each tuple header.
+ This scheme has the advantage of requiring much less space than a design
+ that stores an XID epoch alongside each XID stored in each heap tuple
+ header. It has the disadvantage of constraining the system's ability to
+ allocate new XIDs (in the worst case scenario where transaction ID
+ exhaustion occurs).
+ </para>
+
+ <para>
+ <command>VACUUM</command> will mark tuple headers
+ <emphasis>frozen</emphasis>, indicating that all eligible rows on the
+ page were inserted by a transaction that committed sufficiently far in
+ the past that the effects of the inserting transaction are certain to be
+ visible to all current and future transactions. Normal XIDs are compared
+ using modulo-2<superscript>32</superscript> arithmetic. This means that
+ for every normal XID, there are two billion XIDs that are
+ <quote>older</quote> and two billion that are <quote>newer</quote>;
+ another way to say it is that the normal XID space is circular with no
+ endpoint. Therefore, once a row version has been created with a
+ particular normal XID, the row version will appear to be <quote>in the
+ past</quote> for the next two billion transactions, no matter which
+ normal XID we are talking about. If the row version still exists after
+ more than two billion transactions, it will suddenly appear to be in the
+ future. To prevent this, <productname>PostgreSQL</productname> reserves a
+ special XID, <literal>FrozenTransactionId</literal>, which does not
+ follow the normal XID comparison rules and is always considered older
+ than every normal XID. Frozen row versions are treated as if the
+ inserting XID were <literal>FrozenTransactionId</literal>, so that they
+ will appear to be <quote>in the past</quote> to all normal transactions
+ regardless of wraparound issues, and so such row versions will be valid
+ until deleted, no matter how long that is.
+ </para>
+
+ <note>
+ <para>
+ In <productname>PostgreSQL</productname> versions before 9.4, freezing was
+ implemented by actually replacing a row's insertion XID
+ with <literal>FrozenTransactionId</literal>, which was visible in the
+ row's <structname>xmin</structname> system column. Newer versions just set a flag
+ bit, preserving the row's original <structname>xmin</structname> for possible
+ forensic use. However, rows with <structname>xmin</structname> equal
+ to <literal>FrozenTransactionId</literal> (2) may still be found
+ in databases <application>pg_upgrade</application>'d from pre-9.4 versions.
+ </para>
+ <para>
+ Also, system catalogs may contain rows with <structname>xmin</structname> equal
+ to <literal>BootstrapTransactionId</literal> (1), indicating that they were
+ inserted during the first phase of <application>initdb</application>.
+ Like <literal>FrozenTransactionId</literal>, this special XID is treated as
+ older than every normal XID.
+ </para>
+ </note>
+ </sect3>
+ </sect2>
+
+ <sect2 id="global-transaction-ids">
+ <title>Global Transaction Identifiers</title>
<para>
In addition to <literal>vxid</literal> and <type>xid</type>,
prepared transactions are also assigned Global Transaction
@@ -77,6 +156,7 @@
prepared transactions. The mapping of GID to xid is shown in <link
linkend="view-pg-prepared-xacts"><structname>pg_prepared_xacts</structname></link>.
</para>
+ </sect2>
</sect1>
<sect1 id="xact-locking">
--
2.40.1
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-05-12 01:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-12 17:36 ` Ryan Booz <[email protected]>
2023-05-12 17:37 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
2023-05-13 02:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Ryan Booz @ 2023-05-12 17:36 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
Thanks for the continued work, Peter. I hate to be the guy that starts this way,
but this is my first ever response on pgsql-hackers. (insert awkward
smile face).
Hopefully I've followed etiquette well, but please forgive any
missteps, and I'm
happy for any help in making better contributions in the future.
On Thu, May 11, 2023 at 9:19 PM Peter Geoghegan <[email protected]> wrote:
>
> On Thu, May 4, 2023 at 3:18 PM samay sharma <[email protected]> wrote:
> > What do you think about the term "Exhaustion"? Maybe something like "XID allocation exhaustion" or "Exhaustion of allocatable XIDs"?
>
> I use the term "transaction ID exhaustion" in the attached revision,
> v4. Overall, v4 builds on the work that went into v2 and v3, by
> continuing to polish the overhaul of everything related to freezing,
> relfrozenxid advancement, and anti-wraparound autovacuum.
Just to say on the outset, as has been said earlier in the tread by others,
that this is herculean work. Thank you for putting the effort you have thus far.
There's a lot of good from where I sit in the modification efforts.
It's a heavy,
dense topic, so there's probably never going to be a perfect way to
get it all in,
but some of the context early on, especially, is helpful for framing.
>
> It would be nice if it was possible to add an animation/diagram a
> little like this one: https://tuple-freezing-demo.angusd.com (this is
> how I tend to think about the "transaction ID space".)
Indeed. With volunteer docs, illustrations/diagrams are hard for sure. But,
this or something akin to the "clock" image I've seen elsewhere when
describing the transaction ID space would probably be helpful if it were ever
possible. In fact, there's just a lot about the MVCC stuff in general that
would benefit from diagrams. But alas, I guess that's why we have some
good go-to community talks/slide decks. :-)
> v4 also limits use of the term "wraparound" to places that directly
> discuss anti-wraparound autovacuums (plus one place in xact.sgml,
> where discussion of "true unsigned integer wraparound" and related
> implementation details has been moved). Otherwise we use the term
> "transaction ID exhaustion", which is pretty much the user-facing name
> for "xidStopLimit". I feel that this is a huge improvement, for the
> reason given to Greg earlier. I'm flexible on the details, but I feel
> strongly that we should minimize use of the term wraparound wherever
> it might have the connotation of "the past becoming the future". This
> is not a case of inventing a new terminology for its own sake. If
> anybody is skeptical I ask that they take a look at what I came up
> with before declaring it a bad idea. I have made that as easy as
> possible, by once again attaching a prebuilt routine-vacuuming.html.
Thanks again for doing this. Really helpful for doc newbies like me that
want to help but are still working through the process. Really helpful
and appreciated.
>
>
> Other changes in v4, compared to v3:
>
> * Improved discussion of the differences between non-aggressive and
> aggressive VACUUM.
This was helpful for me and not something I've previously put much thought
into. Helpful context that is missing from the current docs.
> * Explains "catch-up freezing" performed by aggressive VACUUMs directly.
>
> "Catch-up" freezing is the really important "consequence" -- something
> that emerges from how each type of VACUUM behaves over time. It is an
> indirect consequence of the behaviors. I would like to counter the
> perception that some users have about freezing only happening during
> aggressive VACUUMs (or anti-wraparound autovacuums). But more than
> that, talking about catch-up freezing seems essential because it is
> the single most important difference.
>
Similarly, this was helpful overall context of various things
happening with freezing.
> * Much improved handling of the discussion of anti-wraparound
> autovacuum, and how it relates to aggressive VACUUMs, following
> feedback from Samay.
>
> There is now only fairly minimal overlap in the discussion of
> aggressive VACUUM and anti-wraparound autovacuuming. We finish the
> discussion of aggressive VACUUM just after we start discussing
> anti-wraparound autovacuum. This transition works well, because it
> enforces the idea that anti-wraparound autovacuum isn't really special
> compared to any other aggressive autovacuum. This was something that
> Samay expressed particularly concern about: making anti-wraparound
> autovacuums sound less scary. Though it's also a concern I had from
> the outset, based on practical experience and interactions with people
> that have much less knowledge of Postgres than I do.
Agree. This flows fairly well and helps the user understand that each
"next step"
in the vacuum/freezing process has a distinct job based on previous work.
>
> * Anti-wraparound autovacuum is now mostly discussed as something that
> happens to static or mostly-static tables....
> ...This moves discussion of anti-wraparound av in the direction of:
> "Anti-wraparound autovacuum is a special type of autovacuum. Its
> purpose is to ensure that relfrozenxid advances when no earlier VACUUM
> could advance it in passing — often because no VACUUM has run against
> the table for an extended period."
>
Again, learned something new here, at least in how I think about it and talk
with others. In total, I do think these changes make wraparound/exhaustion
seem less "the sky is falling".
> * Added a couple of "Tips" about instrumentation that appears in the
> server log whenever autovacuum reports on a VACUUM operation.
>
> * Much improved "Truncating Transaction Status Information" subsection.
>
> My explanation of the ways in which autovacuum_freeze_max_age can
> affect the storage overhead of commit/abort status in pg_xact is much
> clearer than it was in v3 -- pg_xact truncation is now treated as
> something loosely related to the global config of anti-wraparound
> autovacuum, which makes most sense.
>
This one isn't totally sinking in with me yet. Need another read.
> It took a great deal of effort to find a structure that covered
> everything, and that highlighted all of the important relationships
> without going too far, while at the same time not being a huge mess.
> That's what I feel I've arrived at with v4.
In most respects I agree with the overall flow of changes w.r.t the current doc.
Focusing on all of this as something that should normally just be happening
as part of autovacuum is helpful. Working through it as an order of operations
(and I'm just assuming this is the general order) feels like it ties
things together
a lot more. I honestly come away from this document with more of a "I understand
the process" feel than I did previously.
For now, I'd add the following few comments on the intro section,
2.5.1 and 2.5.2. I
haven't gotten to the bottom sections yet for much feedback.
Intro Comments:
1) "The autovacuum daemon automatically schedules maintenance tasks based on
workload requirements." feels at tension with "Autovacuum scheduling
is controlled
via threshold settings."
Owing to the lingering belief that many users have whereby hosting providers
have magically enabled Postgres to do all of this for you, there is
still a need to
actively deal with these thresholds based on load. That is, as far as
I understand,
Postgres doesn't automatically adjust based on load. Someone/thing
still has to modify
the thresholds as load and data size changes.
If the "workload requirements" is pointing towards aggressive
freezing/wraparound
tasks that happen regardless of thresholds, then for me at least that
isn't clear
in that sentence and it feels like there's an implication that
Postgres/autovacuum
is going to magically adjust overall vacuum work based on database workload.
2) "The intended audience is database administrators that wish to
perform more advanced
autovacuum tuning, with any of the following goals in mind:"
I love calling out the audience in some way. That's really helpful, as are the
stated goals in the bullet list. However, as someone feeling pretty novice
after reading all of this, I can't honestly connect how the content on this page
helps me to more advanced tuning. I have a much better idea how freezing,
in particular, works (yay!), but I'm feeling a bit dense how almost anything
here helps me tune vacuum, at least as it relates to the bullets.
I'm sure you have a connection in mind for each, and certainly understanding the
inner workings of what's happening under the covers is tremendously beneficial,
but when I search for "response" or "performance" in this document, it refers
back to another page (not included in this patch) that talks about the
thresholds.
It might be as simple as adding something to the end of each bullet to draw
that relationship, but as is, it's hard for me to do it mentally (although I can
conjecture a few things on my own)
That said, I definitely appreciate the callout that tuning is an
iterative process
and the minor switch from "creates a substantial amount of I/O
traffic" to "may create...".
** Section 2.5.1 - Recovering Disk Space **
3). "The space dead tuples occupy must eventually be reclaimed for reuse
by new rows, to avoid unbounded growth of disk space requirements. Reclaiming
space from dead rows is VACUUM's main responsibility."
It feels like one connection you could make to the bullet list above
is in this area
and not mentioned. By freeing up space and reducing the number of pages that
need to be read for satisfying a query, vacuum and recovering disk space
(theoretically) improves query performance. Not 100% how to add it in context
of these first two paragraphs.
4) Caution: "It may be a good idea to add monitoring to alert you about this."
I hate to be pedantic about it, but I think we should spell out
"this". Do we have
a pointer in documentation to what kinds of things to monitor for? Am monitoring
long-running transactions or some metric that shows me that VACUUM is being
"held back"? I know what you mean, but it's not clear to me how to do the right
thing in my environment here.
5) The plethora of tips/notes/warnings.
As you and others have mentioned, as presented these really have no context
for me. Individually they are good/helpful information, but it's
really hard to make
a connection to what I should "do" about it.
It seems to me that this would be a good place to put a subsection which is
something like, "A note about reclaiming disk space" or something. In my
experience, most people hear about and end up using VACUUM FULL because
things got out of control and they want to get into a better spot (I have been
in that boat). I think with a small section that says, in essence,
"hey, now that
you understand why/how vacuum reclaims disk resources normally, if you're
in a position where things aren't in a good state, this is what you need to know
if you want to reclaim space from a really inefficient table"
For me, at least, I think it would be easier to read/grok what you're sharing in
these callouts.
6) One last missing piece that very well might be in another page not referenced
(I obviously need to get the PG16 docs pulled and built locally so
that I can have
better overall reference. My apologies).
In my experience, one of the biggest issues with the thresholds and recovering
space is the idea of tuning individual tables, not just the entire
database. 5/10/20%
might be fine for most tables, but it's usually the really active ones
that need the
tuning, specifically lowering the thresholds. That doesn't come across to me in
this section at all. Again, maybe I've missed something on another page and
it's all good, but it felt worth calling out.
Plus, it may provide an opportunity to bring in the threshold formulas again if
they aren't referenced elsewhere (although they probably are).
Hope that makes sense.
** Section 2.5.2: Freezing to manage... **
As stated above, the effort here overall is great IMO. I like the flow
and reduction
in alarmist tone for things like wraparound, etc. I understand more
about freezing,
aggressive and otherwise, than I did before.
7) That said, totally speaking as a non-contributor, this section is
obviously very long
for good reason. But, by the time I've gotten down to 25.2.2.3, my
brain is a bit
bewildered on where we've gotten to. That's more a comment on my capability
to process it all, but I wonder if a slightly more explicit intro
could help set the
stage at least.
"One side-effect of vacuum and transaction ID management at the row level is
that PostgreSQL would normally need to inspect each row for every query to
ensure it is visible to each requesting transaction. In order to
reduce the need to
read and inspect excessive amounts of data at query time or when normal vacuum
maintenance kicks in, VACUUM has a second job called freezing, which
accomplishes three goals: (attempting to tie in the three sections)
* speeding up queries and vacuum operations by...
* advancing the transaction ID space on generally static tables...
* ensure there are always free transaction IDs available for normal
operation...
"
Maybe totally worthless and too much, but something like that might set a reader
up for just a bit more context. Then you could take most of what comes before
"2.5.2.2.1 Aggressive Vacuum" as a subsection (would require a renumber below)
with something like "2.5.2.2.1 Normal Freezing Activity"
8) Note "In PostgreSQL versions before 16..."
Showing my naivety, somehow this isn't connecting with me totally. If
it's important
to call out, then maybe we need a connecting sentence. Based on the content
above, I think you're pointing to "It's also why VACUUM will freeze all eligible
tuples from a heap page once the decision to freeze at least one tuple
is taken:"
If that's it, it's just not clear to me what's totally changed. Sorry,
more learning. :-)
---
Hope something in there is helpful.
Ryan Booz
>
> --
> Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-05-12 01:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-12 17:36 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
@ 2023-05-12 17:37 ` Ryan Booz <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Ryan Booz @ 2023-05-12 17:37 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
And, of course, I forgot that I switch to text-mode after writing most
of this, so the carriage returns were unnecessary. (facepalm... sigh)
--
Ryan
On Fri, May 12, 2023 at 1:36 PM Ryan Booz <[email protected]> wrote:
>
> Thanks for the continued work, Peter. I hate to be the guy that starts this way,
> but this is my first ever response on pgsql-hackers. (insert awkward
> smile face).
> Hopefully I've followed etiquette well, but please forgive any
> missteps, and I'm
> happy for any help in making better contributions in the future.
>
> On Thu, May 11, 2023 at 9:19 PM Peter Geoghegan <[email protected]> wrote:
> >
> > On Thu, May 4, 2023 at 3:18 PM samay sharma <[email protected]> wrote:
> > > What do you think about the term "Exhaustion"? Maybe something like "XID allocation exhaustion" or "Exhaustion of allocatable XIDs"?
> >
> > I use the term "transaction ID exhaustion" in the attached revision,
> > v4. Overall, v4 builds on the work that went into v2 and v3, by
> > continuing to polish the overhaul of everything related to freezing,
> > relfrozenxid advancement, and anti-wraparound autovacuum.
>
> Just to say on the outset, as has been said earlier in the tread by others,
> that this is herculean work. Thank you for putting the effort you have thus far.
> There's a lot of good from where I sit in the modification efforts.
> It's a heavy,
> dense topic, so there's probably never going to be a perfect way to
> get it all in,
> but some of the context early on, especially, is helpful for framing.
>
> >
> > It would be nice if it was possible to add an animation/diagram a
> > little like this one: https://tuple-freezing-demo.angusd.com (this is
> > how I tend to think about the "transaction ID space".)
>
> Indeed. With volunteer docs, illustrations/diagrams are hard for sure. But,
> this or something akin to the "clock" image I've seen elsewhere when
> describing the transaction ID space would probably be helpful if it were ever
> possible. In fact, there's just a lot about the MVCC stuff in general that
> would benefit from diagrams. But alas, I guess that's why we have some
> good go-to community talks/slide decks. :-)
>
> > v4 also limits use of the term "wraparound" to places that directly
> > discuss anti-wraparound autovacuums (plus one place in xact.sgml,
> > where discussion of "true unsigned integer wraparound" and related
> > implementation details has been moved). Otherwise we use the term
> > "transaction ID exhaustion", which is pretty much the user-facing name
> > for "xidStopLimit". I feel that this is a huge improvement, for the
> > reason given to Greg earlier. I'm flexible on the details, but I feel
> > strongly that we should minimize use of the term wraparound wherever
> > it might have the connotation of "the past becoming the future". This
> > is not a case of inventing a new terminology for its own sake. If
> > anybody is skeptical I ask that they take a look at what I came up
> > with before declaring it a bad idea. I have made that as easy as
> > possible, by once again attaching a prebuilt routine-vacuuming.html.
>
> Thanks again for doing this. Really helpful for doc newbies like me that
> want to help but are still working through the process. Really helpful
> and appreciated.
>
> >
> >
> > Other changes in v4, compared to v3:
> >
> > * Improved discussion of the differences between non-aggressive and
> > aggressive VACUUM.
>
> This was helpful for me and not something I've previously put much thought
> into. Helpful context that is missing from the current docs.
>
> > * Explains "catch-up freezing" performed by aggressive VACUUMs directly.
> >
> > "Catch-up" freezing is the really important "consequence" -- something
> > that emerges from how each type of VACUUM behaves over time. It is an
> > indirect consequence of the behaviors. I would like to counter the
> > perception that some users have about freezing only happening during
> > aggressive VACUUMs (or anti-wraparound autovacuums). But more than
> > that, talking about catch-up freezing seems essential because it is
> > the single most important difference.
> >
>
> Similarly, this was helpful overall context of various things
> happening with freezing.
>
> > * Much improved handling of the discussion of anti-wraparound
> > autovacuum, and how it relates to aggressive VACUUMs, following
> > feedback from Samay.
> >
> > There is now only fairly minimal overlap in the discussion of
> > aggressive VACUUM and anti-wraparound autovacuuming. We finish the
> > discussion of aggressive VACUUM just after we start discussing
> > anti-wraparound autovacuum. This transition works well, because it
> > enforces the idea that anti-wraparound autovacuum isn't really special
> > compared to any other aggressive autovacuum. This was something that
> > Samay expressed particularly concern about: making anti-wraparound
> > autovacuums sound less scary. Though it's also a concern I had from
> > the outset, based on practical experience and interactions with people
> > that have much less knowledge of Postgres than I do.
>
> Agree. This flows fairly well and helps the user understand that each
> "next step"
> in the vacuum/freezing process has a distinct job based on previous work.
>
> >
> > * Anti-wraparound autovacuum is now mostly discussed as something that
> > happens to static or mostly-static tables....
> > ...This moves discussion of anti-wraparound av in the direction of:
> > "Anti-wraparound autovacuum is a special type of autovacuum. Its
> > purpose is to ensure that relfrozenxid advances when no earlier VACUUM
> > could advance it in passing — often because no VACUUM has run against
> > the table for an extended period."
> >
>
> Again, learned something new here, at least in how I think about it and talk
> with others. In total, I do think these changes make wraparound/exhaustion
> seem less "the sky is falling".
>
> > * Added a couple of "Tips" about instrumentation that appears in the
> > server log whenever autovacuum reports on a VACUUM operation.
> >
> > * Much improved "Truncating Transaction Status Information" subsection.
> >
> > My explanation of the ways in which autovacuum_freeze_max_age can
> > affect the storage overhead of commit/abort status in pg_xact is much
> > clearer than it was in v3 -- pg_xact truncation is now treated as
> > something loosely related to the global config of anti-wraparound
> > autovacuum, which makes most sense.
> >
> This one isn't totally sinking in with me yet. Need another read.
>
> > It took a great deal of effort to find a structure that covered
> > everything, and that highlighted all of the important relationships
> > without going too far, while at the same time not being a huge mess.
> > That's what I feel I've arrived at with v4.
>
> In most respects I agree with the overall flow of changes w.r.t the current doc.
> Focusing on all of this as something that should normally just be happening
> as part of autovacuum is helpful. Working through it as an order of operations
> (and I'm just assuming this is the general order) feels like it ties
> things together
> a lot more. I honestly come away from this document with more of a "I understand
> the process" feel than I did previously.
>
> For now, I'd add the following few comments on the intro section,
> 2.5.1 and 2.5.2. I
> haven't gotten to the bottom sections yet for much feedback.
>
> Intro Comments:
> 1) "The autovacuum daemon automatically schedules maintenance tasks based on
> workload requirements." feels at tension with "Autovacuum scheduling
> is controlled
> via threshold settings."
>
> Owing to the lingering belief that many users have whereby hosting providers
> have magically enabled Postgres to do all of this for you, there is
> still a need to
> actively deal with these thresholds based on load. That is, as far as
> I understand,
> Postgres doesn't automatically adjust based on load. Someone/thing
> still has to modify
> the thresholds as load and data size changes.
>
> If the "workload requirements" is pointing towards aggressive
> freezing/wraparound
> tasks that happen regardless of thresholds, then for me at least that
> isn't clear
> in that sentence and it feels like there's an implication that
> Postgres/autovacuum
> is going to magically adjust overall vacuum work based on database workload.
>
> 2) "The intended audience is database administrators that wish to
> perform more advanced
> autovacuum tuning, with any of the following goals in mind:"
>
> I love calling out the audience in some way. That's really helpful, as are the
> stated goals in the bullet list. However, as someone feeling pretty novice
> after reading all of this, I can't honestly connect how the content on this page
> helps me to more advanced tuning. I have a much better idea how freezing,
> in particular, works (yay!), but I'm feeling a bit dense how almost anything
> here helps me tune vacuum, at least as it relates to the bullets.
>
> I'm sure you have a connection in mind for each, and certainly understanding the
> inner workings of what's happening under the covers is tremendously beneficial,
> but when I search for "response" or "performance" in this document, it refers
> back to another page (not included in this patch) that talks about the
> thresholds.
>
> It might be as simple as adding something to the end of each bullet to draw
> that relationship, but as is, it's hard for me to do it mentally (although I can
> conjecture a few things on my own)
>
> That said, I definitely appreciate the callout that tuning is an
> iterative process
> and the minor switch from "creates a substantial amount of I/O
> traffic" to "may create...".
>
> ** Section 2.5.1 - Recovering Disk Space **
>
> 3). "The space dead tuples occupy must eventually be reclaimed for reuse
> by new rows, to avoid unbounded growth of disk space requirements. Reclaiming
> space from dead rows is VACUUM's main responsibility."
>
> It feels like one connection you could make to the bullet list above
> is in this area
> and not mentioned. By freeing up space and reducing the number of pages that
> need to be read for satisfying a query, vacuum and recovering disk space
> (theoretically) improves query performance. Not 100% how to add it in context
> of these first two paragraphs.
>
> 4) Caution: "It may be a good idea to add monitoring to alert you about this."
> I hate to be pedantic about it, but I think we should spell out
> "this". Do we have
> a pointer in documentation to what kinds of things to monitor for? Am monitoring
> long-running transactions or some metric that shows me that VACUUM is being
> "held back"? I know what you mean, but it's not clear to me how to do the right
> thing in my environment here.
>
> 5) The plethora of tips/notes/warnings.
> As you and others have mentioned, as presented these really have no context
> for me. Individually they are good/helpful information, but it's
> really hard to make
> a connection to what I should "do" about it.
>
> It seems to me that this would be a good place to put a subsection which is
> something like, "A note about reclaiming disk space" or something. In my
> experience, most people hear about and end up using VACUUM FULL because
> things got out of control and they want to get into a better spot (I have been
> in that boat). I think with a small section that says, in essence,
> "hey, now that
> you understand why/how vacuum reclaims disk resources normally, if you're
> in a position where things aren't in a good state, this is what you need to know
> if you want to reclaim space from a really inefficient table"
>
> For me, at least, I think it would be easier to read/grok what you're sharing in
> these callouts.
>
> 6) One last missing piece that very well might be in another page not referenced
> (I obviously need to get the PG16 docs pulled and built locally so
> that I can have
> better overall reference. My apologies).
>
> In my experience, one of the biggest issues with the thresholds and recovering
> space is the idea of tuning individual tables, not just the entire
> database. 5/10/20%
> might be fine for most tables, but it's usually the really active ones
> that need the
> tuning, specifically lowering the thresholds. That doesn't come across to me in
> this section at all. Again, maybe I've missed something on another page and
> it's all good, but it felt worth calling out.
>
> Plus, it may provide an opportunity to bring in the threshold formulas again if
> they aren't referenced elsewhere (although they probably are).
>
> Hope that makes sense.
>
> ** Section 2.5.2: Freezing to manage... **
> As stated above, the effort here overall is great IMO. I like the flow
> and reduction
> in alarmist tone for things like wraparound, etc. I understand more
> about freezing,
> aggressive and otherwise, than I did before.
>
> 7) That said, totally speaking as a non-contributor, this section is
> obviously very long
> for good reason. But, by the time I've gotten down to 25.2.2.3, my
> brain is a bit
> bewildered on where we've gotten to. That's more a comment on my capability
> to process it all, but I wonder if a slightly more explicit intro
> could help set the
> stage at least.
>
> "One side-effect of vacuum and transaction ID management at the row level is
> that PostgreSQL would normally need to inspect each row for every query to
> ensure it is visible to each requesting transaction. In order to
> reduce the need to
> read and inspect excessive amounts of data at query time or when normal vacuum
> maintenance kicks in, VACUUM has a second job called freezing, which
> accomplishes three goals: (attempting to tie in the three sections)
> * speeding up queries and vacuum operations by...
> * advancing the transaction ID space on generally static tables...
> * ensure there are always free transaction IDs available for normal
> operation...
> "
>
> Maybe totally worthless and too much, but something like that might set a reader
> up for just a bit more context. Then you could take most of what comes before
> "2.5.2.2.1 Aggressive Vacuum" as a subsection (would require a renumber below)
> with something like "2.5.2.2.1 Normal Freezing Activity"
>
> 8) Note "In PostgreSQL versions before 16..."
> Showing my naivety, somehow this isn't connecting with me totally. If
> it's important
> to call out, then maybe we need a connecting sentence. Based on the content
> above, I think you're pointing to "It's also why VACUUM will freeze all eligible
> tuples from a heap page once the decision to freeze at least one tuple
> is taken:"
> If that's it, it's just not clear to me what's totally changed. Sorry,
> more learning. :-)
>
> ---
> Hope something in there is helpful.
>
> Ryan Booz
>
> >
> > --
> > Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-05-12 01:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-12 17:36 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
@ 2023-05-13 02:40 ` Peter Geoghegan <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-13 02:40 UTC (permalink / raw)
To: Ryan Booz <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, May 12, 2023 at 10:36 AM Ryan Booz <[email protected]> wrote:
> Just to say on the outset, as has been said earlier in the tread by others,
> that this is herculean work. Thank you for putting the effort you have thus far.
Thanks!
> > It would be nice if it was possible to add an animation/diagram a
> > little like this one: https://tuple-freezing-demo.angusd.com (this is
> > how I tend to think about the "transaction ID space".)
>
> Indeed. With volunteer docs, illustrations/diagrams are hard for sure. But,
> this or something akin to the "clock" image I've seen elsewhere when
> describing the transaction ID space would probably be helpful if it were ever
> possible. In fact, there's just a lot about the MVCC stuff in general that
> would benefit from diagrams. But alas, I guess that's why we have some
> good go-to community talks/slide decks. :-)
A picture is worth a thousand words. This particular image may be
worth even more, though.
It happens to be *exactly* what I'd have done if I was tasked with
coming up with an animation that conveys the central ideas. Obviously
I brought this image up because I think that it would be great if we
could find a way to do something like that directly (not impossible,
there are a few images already). However, there is a less obvious
reason why I brought it to your attention: it's a very intuitive way
of understanding what I actually intend to convey through words -- at
least as far as talk about the cluster-wide XID space is concerned. It
might better equip you to review the patch series.
Sure, the animation will make the general idea clearer to just about
anybody -- that's a big part of what I like about it. But it also
captures the nuance that might matter to experts (e.g., the oldest XID
moves forward in jerky discrete jumps, while the next/unallocated XID
moves forward in a smooth, continuous fashion). So it works on
multiple levels, for multiple audiences/experience levels, without any
conflicts -- which is no small thing.
Do my words make you think of something a little like the animation?
If so, good.
> Thanks again for doing this. Really helpful for doc newbies like me that
> want to help but are still working through the process. Really helpful
> and appreciated.
I think that this is the kind of thing that particularly benefits from
diversity in perspectives.
> Agree. This flows fairly well and helps the user understand that each
> "next step"
> in the vacuum/freezing process has a distinct job based on previous work.
I'm trying to make it possible to read in short bursts, and to skim.
The easiest wins in this area will come from simply having more
individual sections/headings, and a more consistent structure. The
really difficult part is coming up with prose that can sort of work
for all audiences at the same time -- without alienating anybody.
Here is an example of what I mean:
The general idea of freezing can reasonably be summarized as "a
process that VACUUM uses to make pages self-contained (no need to do
pg_xact lookups anymore), that also has a role in avoiding transaction
ID exhaustion". That is a totally reasonable beginner-level (well,
relative-beginner-level) understanding of freezing. It *isn't* dumbed
down. You, as a beginner, have a truly useful take-away. At the same
time, you have avoided learning anything that you'll need to unlearn
some day. If I can succeed in doing that, I'll feel a real sense of
accomplishment.
> > * Much improved "Truncating Transaction Status Information" subsection.
> >
> > My explanation of the ways in which autovacuum_freeze_max_age can
> > affect the storage overhead of commit/abort status in pg_xact is much
> > clearer than it was in v3 -- pg_xact truncation is now treated as
> > something loosely related to the global config of anti-wraparound
> > autovacuum, which makes most sense.
> >
> This one isn't totally sinking in with me yet. Need another read.
"Truncating Transaction Status Information" is explicitly supposed to
matter much less than the rest of the stuff on freezing. The main
benefit that the DBA can expect from understanding this content is how
to save a few GB of disk space for pg_xact, which isn't particularly
likely to be possible, and is very very unlikely to be of any real
consequence, compared to everything else. If you were reading the
revised "Routine Vacuuming" as the average DBA, what you'd probably
have ended up doing is just not reading this part at all. And that
would probably be the ideal outcome. It's roughly the opposite of what
you'll get right now, by the way (bizarrely, the current docs place a
great deal of emphasis on this).
(Of course I welcome your feedback here too. Just giving you the context.)
> > It took a great deal of effort to find a structure that covered
> > everything, and that highlighted all of the important relationships
> > without going too far, while at the same time not being a huge mess.
> > That's what I feel I've arrived at with v4.
>
> In most respects I agree with the overall flow of changes w.r.t the current doc.
> Focusing on all of this as something that should normally just be happening
> as part of autovacuum is helpful. Working through it as an order of operations
> (and I'm just assuming this is the general order) feels like it ties
> things together
> a lot more. I honestly come away from this document with more of a "I understand
> the process" feel than I did previously.
That's great news. It might be helpful to give you more context about
the particular approach I've taken here, and how it falls short of
what I'd ideally like to do, in my own mind.
There are some rather strange things that happen to be true about
VACUUM and freezing today, that definitely influenced the way I
structured the docs. I can imagine an improved version of VACUUM that
is not so different to the real VACUUM that we have today (one that
still has freezing as we know it), that still has a much simpler UI --
some policy-based process for deciding which pages to freeze that was
much smarter than a simple trigger. If we were living in a world where
VACUUM actually worked like that, then I'd have been able to come up
with a structure that is a lot closer to what you might have been
hoping for from this patch series. At the very least, I'd have been
able to add some "TL;DR" text at the start of each section, that just
gave the main practical takeaway.
Take vacuum_freeze_min_age. It's a *really* bad design, even on its
own terms, even if we assume that nothing can change about how
freezing works. Yet it's probably still the most important
freezing-related GUC, even in Postgres 16. History matters here. The
GUC was invented in a world before the visibility map existed. When
the visibility map was invented, aggressive VACUUM was also invented
(before then the name for what we now call "aggressive VACUUM" was
actually just "VACUUM"). This development utterly changed the way that
vacuum_freeze_min_age actually works, but we still talk about it as if
its idea of "age" can be considered in isolation, as a universal
catch-all that can be tuned iteratively. The reality is that it is
interpreted in a way that is *hopelessly* tied to other things.
This isn't a minor point. There are really bizarre implications, with
real practical consequences. For example, suppose you want to make
autovacuums run more often against a simple append-only table -- so
you lower autovacuum_vacuum_insert_scale_factor with that in mind.
It's entirely possible that you'll now do *less* useful work, even
though you specifically set out to vacuum more aggressively! This is
due to the way the GUCs interact with each other, of course: the more
often VACUUM runs, the less likely it is that it'll find XIDs before
vacuum_freeze_min_age to trigger freezing during any individual VACUUM
operation, the less useful work you'll do (you'll just accumulate
unfrozen all-visible pages until you finally have an aggressive
VACUUM).
This is exactly as illogical as it sounds. Postgres 16 will be the
first version that even shows instrumentation around freezing at all
in the log reports from autovacuum. This will be a real eye-opener, I
suspect -- I predict that people will be surprised at how freezing
works with their workload, when they finally have the opportunity to
see it for themselves.
> Owing to the lingering belief that many users have whereby hosting providers
> have magically enabled Postgres to do all of this for you, there is
> still a need to
> actively deal with these thresholds based on load. That is, as far as
> I understand,
> Postgres doesn't automatically adjust based on load. Someone/thing
> still has to modify
> the thresholds as load and data size changes.
Well, vacuum_freeze_min_age (anything based on XID age) runs into the
following problem: what is the relationship between XID age, and
freezing work? This is a question whose answer is much too
complicated, suggesting that it's just the wrong question. There is
absolutely no reason to expect a linear relationship (or anything like
it) between XIDs consumed and WAL required to freeze rows from those
XIDs. It's a totally chaotic thing.
The reason for this is: of course it is, why wouldn't it be? On Monday
you'll do a bulk load, and 1 XID will write 1TB to one table. On
Tuesday, there might be only one row per XID consumed, with millions
and millions of rows inserted. This is 100% common sense, and yet is
kinda at odds with the whole idea of basing the decision to freeze on
age (as if vacuum_freeze_min_age didn't have enough problems
already!).
For now, I think our best bet is to signal the importance of avoiding
disaster to intermediate users, and signal the importance of iterative
tuning to advanced users.
> If the "workload requirements" is pointing towards aggressive
> freezing/wraparound
> tasks that happen regardless of thresholds, then for me at least that
> isn't clear
> in that sentence and it feels like there's an implication that
> Postgres/autovacuum
> is going to magically adjust overall vacuum work based on database workload.
That's a good point.
> 2) "The intended audience is database administrators that wish to
> perform more advanced
> autovacuum tuning, with any of the following goals in mind:"
>
> I love calling out the audience in some way. That's really helpful, as are the
> stated goals in the bullet list. However, as someone feeling pretty novice
> after reading all of this, I can't honestly connect how the content on this page
> helps me to more advanced tuning.
You're right to point that out; the actual content here was written
half-heartedly, in part because it depends on the dead-tuple-space
patch, which is not my focus at all right now.
Here is what I'd like the message to be, roughly:
1. This isn't something that you read once. You read it in small
bites. You come back to it from time to time (or you will if you need
to).
At one point Samay said: "I'll give my perspective as someone who has
not read the vacuum code but have learnt most of what I know about
autovacuum / vacuuming by reading the "Routine Vacuuming" page 10s of
times". I fully expect that a minority of users will want to do the
same with these revised docs. The content is very much not supposed to
be read through in one sitting (not if you expect to get any value out
of it). It is very calorie dense, and I don't think that that's really
a problem to be solved.
You have a much better chance of getting value out of it if you as a
user refer back to it as problems emerge. Some things may only click
after the second or third read, based on the experience of trying to
put something else into action in production.
2. If you don't like that it's calorie dense, then that's probably
okay -- just don't read past the parts that seem useful.
3. There are one or two exceptions (e.g., the "Tip" about freezing for
append-only tables), but overall there isn't going to be a simple
formula to follow -- the closest thing might be "don't bother doing
anything until it proves necessary".
This is because too much depends on individual workload requirements.
It is also partly due to it just being really hard to tune things like
vacuum_freeze_min_age very well right now.
4. It's an applied process. The emphasis should be on solving
practical, observed problems that are directly observed -- this isn't
a cookbook (though there are a couple of straightforward recipes,
covering one or two specific things).
> ** Section 2.5.1 - Recovering Disk Space **
It should be noted that what I've done in this area is quite
incomplete. I have only really done structural things here, and some
of these may not be much good.
> It feels like one connection you could make to the bullet list above
> is in this area
> and not mentioned. By freeing up space and reducing the number of pages that
> need to be read for satisfying a query, vacuum and recovering disk space
> (theoretically) improves query performance. Not 100% how to add it in context
> of these first two paragraphs.
It's hard, because it's not so much that vacuuming improves query
performance. It's more like *not* vacuuming hurts it. The exact point
that it starts to hurt is rather hard to predict -- and there might be
little point in trying to predict it with precision.
I tend to think that I'd probably be better off saying nothing about
query response times. Or saying something negative (what to definitely
avoid), not something positive (what to do) -- I would expect it to
generalize a lot better that way.
> 4) Caution: "It may be a good idea to add monitoring to alert you about this."
> I hate to be pedantic about it, but I think we should spell out
> "this". Do we have
> a pointer in documentation to what kinds of things to monitor for? Am monitoring
> long-running transactions or some metric that shows me that VACUUM is being
> "held back"? I know what you mean, but it's not clear to me how to do the right
> thing in my environment here.
Will do.
> 5) The plethora of tips/notes/warnings.
> As you and others have mentioned, as presented these really have no context
> for me. Individually they are good/helpful information, but it's
> really hard to make
> a connection to what I should "do" about it.
Yeah, I call that out in the relevant commit message of the patch as
bad, as temporary.
> It seems to me that this would be a good place to put a subsection which is
> something like, "A note about reclaiming disk space" or something. In my
> experience, most people hear about and end up using VACUUM FULL because
> things got out of control and they want to get into a better spot (I have been
> in that boat). I think with a small section that says, in essence,
> "hey, now that
> you understand why/how vacuum reclaims disk resources normally, if you're
> in a position where things aren't in a good state, this is what you need to know
> if you want to reclaim space from a really inefficient table"
>
> For me, at least, I think it would be easier to read/grok what you're sharing in
> these callouts.
That's the kind of thing that I had planned on with VACUUM FULL,
actually. You know, once I'm done with freezing. There is passing
mention of this in the relevant commit message.
> In my experience, one of the biggest issues with the thresholds and recovering
> space is the idea of tuning individual tables, not just the entire
> database. 5/10/20%
> might be fine for most tables, but it's usually the really active ones
> that need the
> tuning, specifically lowering the thresholds. That doesn't come across to me in
> this section at all. Again, maybe I've missed something on another page and
> it's all good, but it felt worth calling out.
I think that that's true. The rules are kind of different for larger tables.
> read and inspect excessive amounts of data at query time or when normal vacuum
> maintenance kicks in, VACUUM has a second job called freezing, which
> accomplishes three goals: (attempting to tie in the three sections)
> * speeding up queries and vacuum operations by...
> * advancing the transaction ID space on generally static tables...
> * ensure there are always free transaction IDs available for normal
> operation...
> "
>
> Maybe totally worthless and too much, but something like that might set a reader
> up for just a bit more context. Then you could take most of what comes before
> "2.5.2.2.1 Aggressive Vacuum" as a subsection (would require a renumber below)
> with something like "2.5.2.2.1 Normal Freezing Activity"
I think I know what you mean. What I've tried to do here is start with
freezing, and describe it as something that has immediate benefits,
that can be understood as useful, independently of its role in
advancing relfrozenxid later on. So now you wonder: what specific
benefits do I get?
It's hard to be too concrete about those benefits, because you have
things like hint bits. I could say something like that, but I think
I'd have to hedge too much, because you also have hint bits, that help
query response times in roughly the same way (albeit less reliably,
albeit without being set on physical replication standbys when they're
set on the primary).
> 8) Note "In PostgreSQL versions before 16..."
> Showing my naivety, somehow this isn't connecting with me totally. If
> it's important
> to call out, then maybe we need a connecting sentence. Based on the content
> above, I think you're pointing to "It's also why VACUUM will freeze all eligible
> tuples from a heap page once the decision to freeze at least one tuple
> is taken:"
> If that's it, it's just not clear to me what's totally changed. Sorry,
> more learning. :-)
In Postgres 15, vacuum_freeze_min_age was applied in a way that only
froze whatever XIDs could be frozen from the page -- so if you had
half the tuples that were older, and half that were younger, you'd
only freeze the older half. Even when it might have cost you
practically nothing to freeze them all in one go. Now, as the text
you've quoted points out, vacuum_freeze_min_age triggers freezing at
the level of whole pages, including for new XIDs (though only if
they're eligible to be frozen, meaning that everybody agrees that
they're all visible now). So vacuum_freeze_min_age picks pages to
freeze, not individual tuples to freeze (this optimization is so
obvious that it's a little surprising that it took as long as it did
to get in).
Page-level freezing justifies the following statement from the patch,
for example:
"It doesn't matter if it was vacuum_freeze_table_age or
vacuum_multixact_freeze_table_age that made VACUUM use its aggressive
strategy. Every aggressive VACUUM will advance relfrozenxid and
relminmxid by applying the same generic policy that controls which
pages are frozen."
Now, since freezing works at the level of physical heap pages in 16,
the thing that triggers aggressive VACUUM matters less (just as the
thing that triggers freezing of individual pages matters much less --
freezing is freezing). There is minimal risk of freezing the same page
3 times during each of 3 different aggressive VACUUMs. To a much
greater extent, 3 aggressive VACUUMs isn't that different to only 1
aggressive VACUUM for those pages that were already "settled" from the
start. As a result, the addition of page-level freezing made
vacuum_freeze_min_age somewhat less bad -- in 16, its behavior was a
little less dependent on the phase of the moon (especially during
aggressive VACUUMs).
I really value stuff like that -- cases where you as a user can think
of something as independent to some other thing that you also need to
tune. There needs to be a lot more such improvements, but at least we
have this one now.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-11 20:04 ` Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Greg Stark @ 2023-05-11 20:04 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 3 May 2023 at 18:50, Peter Geoghegan <[email protected]> wrote:
>
> What about "XID allocation overload"? The implication that I'm going
> for here is that the system was misconfigured, or there was otherwise
> some kind of imbalance between XID supply and demand.
Fwiw while "wraparound" has pitfalls I think changing it for a new
word isn't really helpful. Especially if it's a mostly meaningless
word like "overload" or "exhaustion". It suddenly makes every existing
doc hard to find and confusing to read.
I say "exhaustion" or "overload" are meaningless because their meaning
is entirely dependent on context. It's not like memory exhaustion or
i/o overload where it's a finite resource and it's just the sheer
amount in use that matters. One way or another the user needs to
understand that it's two numbers marching through a sequence and the
distance between them matters.
I feel like "wraparound" while imperfect is not any worse than any
other word. It still requires context to understand but it's context
that there are many docs online that already explain and are
googleable.
If we wanted a new word it would be "overrun" but like I say, it would
just create a new context dependent technical term that users would
need to find docs that explain and give context. I don't think that
really helps users at all
--
greg
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
@ 2023-05-11 20:40 ` Peter Geoghegan <[email protected]>
2023-05-11 20:49 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-14 02:46 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-11 20:40 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, May 11, 2023 at 1:04 PM Greg Stark <[email protected]> wrote:
> Fwiw while "wraparound" has pitfalls I think changing it for a new
> word isn't really helpful. Especially if it's a mostly meaningless
> word like "overload" or "exhaustion". It suddenly makes every existing
> doc hard to find and confusing to read.
Just to be clear, I am not proposing changing the name of
anti-wraparound autovacuum at all. What I'd like to do is use a term
like "XID exhaustion" to refer to the state that we internally refer
to as xidStopLimit. My motivation is simple: we've completely
terrified users by emphasizing wraparound, which is something that is
explicitly and prominently presented as a variety of data corruption.
The docs say this:
"But since transaction IDs have limited size (32 bits) a cluster that
runs for a long time (more than 4 billion transactions) would suffer
transaction ID wraparound: the XID counter wraps around to zero, and
all of a sudden transactions that were in the past appear to be in the
future — which means their output become invisible. In short,
catastrophic data loss."
> I say "exhaustion" or "overload" are meaningless because their meaning
> is entirely dependent on context. It's not like memory exhaustion or
> i/o overload where it's a finite resource and it's just the sheer
> amount in use that matters.
But transaction IDs are a finite resource, in the sense that you can
never have more than about 2.1 billion distinct unfrozen XIDs at any
one time. "Transaction ID exhaustion" is therefore a lot more
descriptive of the underlying problem. It's a lot better than
wraparound, which, as I've said, is inaccurate in two major ways:
1. Most cases involving xidStopLimit (or even single-user mode data
corruption) won't involve any kind of physical integer wraparound.
2. Most physical integer wraparound is harmless and perfectly routine.
But even this is fairly secondary to me. I don't actually think it's
that important that the name describe exactly what's going on here --
that's expecting rather a lot from a name. That's not really the goal.
The goal is to undo the damage of documentation that heavily implies
that data corruption is the eventual result of not doing enough
vacuuming, in its basic introductory remarks to freezing stuff.
Like Samay, my consistent experience (particularly back in my Heroku
days) has been that people imagine that data corruption would happen
when the system reached what we'd call xidStopLimit. Can you blame
them for thinking that? Almost any name for xidStopLimit that doesn't
have that historical baggage seems likely to be a vast improvement.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-11 20:49 ` Peter Geoghegan <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-11 20:49 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, May 11, 2023 at 1:40 PM Peter Geoghegan <[email protected]> wrote:
> Just to be clear, I am not proposing changing the name of
> anti-wraparound autovacuum at all. What I'd like to do is use a term
> like "XID exhaustion" to refer to the state that we internally refer
> to as xidStopLimit. My motivation is simple: we've completely
> terrified users by emphasizing wraparound, which is something that is
> explicitly and prominently presented as a variety of data corruption.
> The docs say this:
>
> "But since transaction IDs have limited size (32 bits) a cluster that
> runs for a long time (more than 4 billion transactions) would suffer
> transaction ID wraparound: the XID counter wraps around to zero, and
> all of a sudden transactions that were in the past appear to be in the
> future — which means their output become invisible. In short,
> catastrophic data loss."
Notice that this says that "catastrophic data loss" occurs when "the
XID counter wraps around to zero". I think that this was how it worked
before the invention of freezing, over 20 years ago -- the last time
the system would allocate about 4 billion XIDs without doing any
freezing.
While it is still possible to corrupt the database in single user
mode, it has precisely nothing to do with the point that "the XID
counter wraps around to zero". I believe that this wording has done
not insignificant damage to the project's reputation. But let's assume
for a moment that there's only a tiny chance that I'm right about all
of this -- let's assume I'm probably just being alarmist about how
this has been received in the wider world. Even then: why take even a
small chance?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-14 02:46 ` Greg Stark <[email protected]>
2023-05-14 20:59 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Greg Stark @ 2023-05-14 02:46 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 11 May 2023 at 16:41, Peter Geoghegan <[email protected]> wrote:
>
> > I say "exhaustion" or "overload" are meaningless because their meaning
> > is entirely dependent on context. It's not like memory exhaustion or
> > i/o overload where it's a finite resource and it's just the sheer
> > amount in use that matters.
>
> But transaction IDs are a finite resource, in the sense that you can
> never have more than about 2.1 billion distinct unfrozen XIDs at any
> one time. "Transaction ID exhaustion" is therefore a lot more
> descriptive of the underlying problem. It's a lot better than
> wraparound, which, as I've said, is inaccurate in two major ways:
I realize that's literally true that xids are a finite resource. But
that's not what people think of when you talk about exhausting a
finite resource.
It's not like there are 2 billion XIDs in a big pool being used and
returned and as long as you don't use too many XIDs leaving the pool
empty you're ok. When people talk about resource exhaustion they
imagine that they just need a faster machine or some other way of just
putting more XIDs in the pool so they can keep using them at a faster
rate.
I really think focusing on changing one term of art for another,
neither of which is at all meaningful without an extensive technical
explanation helps anyone. All it does is hide all the existing
explanations that are all over the internet from them.
> 1. Most cases involving xidStopLimit (or even single-user mode data
> corruption) won't involve any kind of physical integer wraparound.
Fwiw I've never actually bumped into anyone talking about integer
overflow (which isn't usually called "wraparound" anyways). And in any
case it's not a terrible misconception, it at least gives users a
reasonable model for how XID space consumption works. In fact it's not
even entirely wrong -- it's not the XID itself that's overflowing but
the difference between the XID and the frozen XID.
--
greg
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-14 02:46 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
@ 2023-05-14 20:59 ` Peter Geoghegan <[email protected]>
2023-05-15 03:10 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-14 20:59 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, May 13, 2023 at 7:47 PM Greg Stark <[email protected]> wrote:
> It's not like there are 2 billion XIDs in a big pool being used and
> returned and as long as you don't use too many XIDs leaving the pool
> empty you're ok. When people talk about resource exhaustion they
> imagine that they just need a faster machine or some other way of just
> putting more XIDs in the pool so they can keep using them at a faster
> rate.
>
> I really think focusing on changing one term of art for another,
> neither of which is at all meaningful without an extensive technical
> explanation helps anyone. All it does is hide all the existing
> explanations that are all over the internet from them.
Have you read the documentation in question recently? The first two
paragraphs, in particular:
https://www.postgresql.org/docs/devel/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
As I keep pointing out, we literally introduce the whole topic of
freezing/wraparound by telling users that VACUUM needs to avoid
wraparound to stop your database from becoming corrupt. Which is when
"the past becomes the future", or in simple terms data corruption
(without any qualification about single user mode being required to
corrupt the DB). Users think that that's what "wraparound" means
because we've taught them to think that. We've already been giving
users "an extensive technical explanation" for many years -- one that
happens to be both harmful and factually incorrect.
I agree that users basically don't care about unsigned vs signed vs
whatever. But there is a sense that that matters, because the docs
have made it matter. That's my starting point. That's the damage that
I'm trying to undo.
> > 1. Most cases involving xidStopLimit (or even single-user mode data
> > corruption) won't involve any kind of physical integer wraparound.
>
> Fwiw I've never actually bumped into anyone talking about integer
> overflow (which isn't usually called "wraparound" anyways). And in any
> case it's not a terrible misconception, it at least gives users a
> reasonable model for how XID space consumption works. In fact it's not
> even entirely wrong -- it's not the XID itself that's overflowing but
> the difference between the XID and the frozen XID.
Even your "not entirely wrong" version is entirely wrong. What you
describe literally cannot happen (outside of single user mode),
because xidStopLimit stops it from happening. To me your argument
seems similar to arguing that it's okay to call chemotherapy "cancer"
on the grounds that "cancer" refers to something that you really ought
to avoid in the first place in any case, which makes the whole
distinction irrelevant to non-oncologists.
That said, I concede that the term wraparound is too established to
just get rid of now. The only viable way forward now may be to
encourage users to think about it in the way that you suppose they
must already think about it. That is, to prominently point out that
"wraparound" actually refers to a protective mode of operation where
XID allocations are temporarily disallowed. And not pretty much
nothing to do with "wraparound" of the kind that the user may be
familiar with from other contexts. Including (and especially) all
earlier versions of the Postgres docs.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-14 02:46 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-14 20:59 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-15 03:10 ` Peter Geoghegan <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Peter Geoghegan @ 2023-05-15 03:10 UTC (permalink / raw)
To: Greg Stark <[email protected]>; +Cc: samay sharma <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sun, May 14, 2023 at 1:59 PM Peter Geoghegan <[email protected]> wrote:
> Have you read the documentation in question recently? The first two
> paragraphs, in particular:
>
> https://www.postgresql.org/docs/devel/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
>
> As I keep pointing out, we literally introduce the whole topic of
> freezing/wraparound by telling users that VACUUM needs to avoid
> wraparound to stop your database from becoming corrupt. Which is when
> "the past becomes the future"
I went through the history of maintenance.sgml. "Routine Vacuuming"
dates back to 2001. Sure enough, our current "25.1.5. Preventing
Transaction ID Wraparound Failures" introductory paragraphs (the ones
that I find so misleading and alarmist) appear in the original
version, too. But in 2001, they weren't alarmist -- they were
proportionate to the risk that existed at the time. This becomes
totally clear once you see the original. In particular, once you see
the current introductory paragraphs next to another paragraph in the
original 2001 version:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=doc/src/sgml/maintenance.sgml;h=7629c5b...
The later paragraph follows up by saying: "In practice this [somewhat
regular vacuuming] isn't an onerous requirement, but since the
consequences of failing to meet it can be ___complete data loss___
(not just wasted disk space or slow performance), some special
provisions...". This means that my particular interpretation of the
25.1.5. introductory paragraphs are absolutely consistent with the
original intent from the time they were written. I'm now more
confident than ever that all of the stuff about "catastrophic data
loss" should have been removed in 2005 or 2006 at the latest. It
*almost* was removed around that time, but for whatever reason it
wasn't removed in full. And for whatever reason it didn't quite
register with anybody in a position to do much about it.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
@ 2023-05-04 21:57 ` samay sharma <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: samay sharma @ 2023-05-04 21:57 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
On Wed, May 3, 2023 at 2:59 PM Peter Geoghegan <[email protected]> wrote:
> Hi Samay,
>
> On Tue, May 2, 2023 at 11:40 PM samay sharma <[email protected]>
> wrote:
> > Thanks for taking the time to do this. It is indeed difficult work.
>
> Thanks for the review! I think that this is something that would
> definitely benefit from a perspective such as yours.
>
Glad to hear that my feedback was helpful.
>
> > There are things I like about the changes you've proposed and some where
> I feel that the previous section was easier to understand.
>
> That makes sense, and I think that I agree with every point you've
> raised, bar none. I'm pleased to see that you basically agree with the
> high level direction.
>
> I would estimate that the version you looked at (v2) is perhaps 35%
> complete. So some of the individual problems you noticed were a direct
> consequence of the work just not being anywhere near complete. I'll
> try to do a better job of tracking the relative maturity of each
> commit/patch in each commit message, going forward.
>
> Anything that falls under "25.2.1. Recovering Disk Space" is
> particularly undeveloped in v2. The way that I broke that up into a
> bunch of WARNINGs/NOTEs/TIPs was just a short term way of breaking it
> up into pieces, so that the structure was very approximately what I
> wanted. I actually think that the stuff about CLUSTER and VACUUM FULL
> belongs in a completely different chapter. Since it is not "Routine
> Vacuuming" at all.
>
> >> 2. Renamed "Preventing Transaction ID Wraparound Failures" to
> >> "Freezing to manage the transaction ID space". Now we talk about
> >> wraparound as a subtopic of freezing, not vice-versa. (This is a
> >> complete rewrite, as described by later items in this list).
> >
> > +1 on this too. Freezing is a normal part of vacuuming and while the
> aggressive vacuums are different, I think just talking about the worst case
> scenario while referring to it is alarmist.
>
> Strangely enough, Postgres 16 is the first version that instruments
> freezing in its autovacuum log reports. I suspect that some long term
> users will find it quite surprising to see how much (or how little)
> freezing takes place in non-aggressive VACUUMs.
>
> The introduction of page-level freezing will make it easier and more
> natural to tune settings like vacuum_freeze_min_age, with the aim of
> smoothing out the burden of freezing over time (particularly by making
> non-aggressive VACUUMs freeze more). Page-level freezing removes any
> question of not freezing every tuple on a page (barring cases where
> "removable cutoff" is noticeably held back by an old MVCC snapshot).
> This makes it more natural to think of freezing as a process that
> makes it okay to store data in individual physical heap pages, long
> term.
>
> > 1) While I agree that bundling VACUUM and VACUUM FULL is not the right
> way, moving all VACUUM FULL references into tips and warnings also seems
> excessive. I think it's probably best to just have a single paragraph which
> talks about VACUUM FULL as I do think it should be mentioned in the
> reclaiming disk space section.
>
> As I mentioned briefly already, my intention is to move it to another
> chapter entirely. I was thinking of "Chapter 29. Monitoring Disk
> Usage". The "Routine Vacuuming" docs would then link to this sect1 --
> something along the lines of "non-routine commands to reclaim a lot of
> disk space in the event of extreme bloat".
>
> > 2) I felt that the new section, "Freezing to manage the transaction ID
> space" could be made simpler to understand. As an example, I understood
> what the parameters (autovacuum_freeze_max_age, vacuum_freeze_table_age) do
> and how they interact better in the previous version of the docs.
>
> Agreed. I'm going to split it up some more. I think that the current
> "25.2.2.1. VACUUM's Aggressive Strategy" should be split in two, so we
> go from talking about aggressive VACUUMs to Antiwraparound
> autovacuums. Finding the least confusing way of explaining it has been
> a focus of mine in the last few days.
>
To be honest, this was not super simple to understand even in the previous
version. However, as our goal is to simplify this and make it easier to
understand, I'll hold this patch-set to a higher standard :).
I wish there was a simple representation (maybe even a table or something)
which would explain the differences between a VACUUM which is not
aggressive, a VACUUM which ends up being aggressive due to
vacuum_freeze_table_age and an antiwraparound autovacuum.
>
> > 4) I think we should explicitly call out that seeing an anti-wraparound
> VACUUM or "VACUUM table (to prevent wraparound)" is normal and that it's
> just a VACUUM triggered due to the table having unfrozen rows with an XID
> older than autovacuum_freeze_max_age. I've seen many users panicking on
> seeing this and feeling that they are close to a wraparound.
>
> That has also been my exact experience. Users are terrified, usually
> for no good reason at all. I'll make sure that this comes across in
> the next revision of the patch series.
>
Thinking about it a bit more, I wonder if there's value in changing the
"(to prevent wraparound)" to something else. It's understandable why people
who just see that in pg_stat_activity and don't read docs might assume they
are close to a wraparound.
Regards,
Samay
>
> > Also, we should be more clear about how it's different from VACUUMs
> triggered due to the scale factors (cancellation behavior, being triggered
> when autovacuum is disabled etc.).
>
> Right. Though I think that the biggest point of confusion for users is
> how *few* differences there really are between antiwraparound
> autovacuum, and any other kind of autovacuum that happens to use
> VACUUM's aggressive strategy. There is really only one important
> difference: the autocancellation behavior. This is an autovacuum
> behavior, not a VACUUM behavior -- so the "VACUUM side" doesn't know
> anything about that at all.
> > 5) Can we use a better name for the XidStopLimit mode? It seems like a
> very implementation centric name. Maybe a better version of "Running out of
> the XID space" or something like that?
>
> Coming up with a new user-facing name for xidStopLimit is already on
> my TODO list (it's surprisingly hard). I have used that name so far
> because it unambiguously refers to the exact thing that I want to talk
> about when discussing the worst case. Other than that, it's a terrible
> name.
>
> > 6) In the XidStopLimit mode section, it would be good to explain briefly
> why you could get to this scenario. It's not something which should happen
> in a normal running system unless you have a long running transaction or
> inactive replication slots or a badly configured system or something of
> that sort.
>
> I agree that that's important. Note that there is already something
> about "removable cutoff" being held back at the start of the
> discussion of freezing -- that will prevent freezing in exactly the
> same way as it prevents cleanup of dead tuples.
>
> That will become a WARNING box in the next revision. There should also
> be a similar, analogous WARNING box (about "removable cutoff" being
> held back) much earlier on in the docs -- this should appear in
> "25.2.1. Recovering Disk Space". Obviously this structure suggests
> that there is an isomorphism between freezing and removing bloat. For
> example, if you cannot "freeze" an XID that appears in some tuple's
> xmax, then you also cannot remove that tuple because VACUUM only sees
> it as a recently dead tuple (if xmax is >= OldestXmin/removable
> cutoff, and from a deleter that already committed).
>
> I don't think that we need to spell the "isomorphism" point out to the
> reader directly, but having a subtle cue that that's how it works
> seems like a good idea.
>
> > If you got to this point, other than running VACUUM to get out of the
> situation, it's also important to figure out what got you there in the
> first place as many VACUUMs should have attempted to advance the
> relfrozenxid and failed.
>
> It's also true that problems that can lead to the system entering
> xidStopLimit mode aren't limited to cases where doing required
> freezing is fundamentally impossible due to something holding back
> "removable cutoff". It's also possible that VACUUM simply can't keep
> up (though the failsafe has helped with that problem a lot).
>
> I tend to agree that there needs to be more about this in the
> xidStopLimit subsection (discussion of freezing being held back by
> "removable cutoff" is insufficient), but FWIW that seems like it
> should probably be treated as out of scope for this patch. It is more
> the responsibility of the other patch [1] that aims to put the
> xidStopLimit documentation on a better footing (and remove that
> terrible HINT about single user mode).
>
> Of course, that other patch is closely related to this patch -- the
> precise boundaries are unclear at this point. In any case I think that
> this should happen, because I think that it's a good idea.
>
> > There are a few other small things I noticed along the way but my goal
> was to look at the overall structure.
>
> Thanks again! This is very helpful.
>
> [1]
> https://www.postgresql.org/message-id/flat/CAJ7c6TM2D277U2wH8X78kg8pH3tdUqebV3_JCJqAkYQFHCFzeg%40mai...
> --
> Peter Geoghegan
>
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 7/7] Allow to print raw parse tree.
@ 2023-07-26 10:49 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cc99ec9c..c01e90f735 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Wed_Jul_26_21_21_34_2023_317)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v4 7/7] Allow to print raw parse tree.
@ 2023-08-09 07:56 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cc99ec9c..c01e90f735 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Wed_Aug__9_17_41_12_2023_134)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v5 7/7] Allow to print raw parse tree.
@ 2023-09-02 06:32 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e4756f8be2..fc8efa915b 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Sat_Sep__2_15_52_35_2023_273)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v6 7/7] Allow to print raw parse tree.
@ 2023-09-12 05:22 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21b9763183..3e3653816e 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -651,6 +651,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Tue_Sep_12_15_18_43_2023_359)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v7 7/7] Allow to print raw parse tree.
@ 2023-09-22 04:53 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21b9763183..3e3653816e 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -651,6 +651,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Fri_Sep_22_14_16_40_2023_530)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v8 7/7] Allow to print raw parse tree.
@ 2023-09-25 05:01 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21b9763183..3e3653816e 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -651,6 +651,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Mon_Sep_25_14_26_30_2023_752)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v9 7/7] Allow to print raw parse tree.
@ 2023-10-04 05:51 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21b9763183..3e3653816e 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -651,6 +651,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Wed_Oct__4_15_03_28_2023_821)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v10 7/7] Allow to print raw parse tree.
@ 2023-10-22 02:22 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index c900427ecf..fa5d862604 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -652,6 +652,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Sun_Oct_22_11_39_20_2023_140)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v11 7/7] Allow to print raw parse tree.
@ 2023-11-08 06:57 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 6a070b5d8c..beb528f526 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -652,6 +652,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Wed_Nov__8_16_37_05_2023_872)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v12 7/7] Allow to print raw parse tree.
@ 2023-12-04 11:23 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2023-12-04 11:23 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7298a187d1..b43afad0be 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -652,6 +652,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Fri_Dec__8_10_16_13_2023_489)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v13 8/8] Allow to print raw parse tree.
@ 2024-01-22 09:45 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 1a34bd3715..68f5cf3667 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -652,6 +652,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Mon_Jan_22_19_26_18_2024_011)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v14 8/8] Allow to print raw parse tree.
@ 2024-02-28 13:59 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-02-28 13:59 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 59ab812d2e..e433ddafe0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -652,6 +652,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Thu_Feb_29_09_19_54_2024_640)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v15 8/8] Allow to print raw parse tree.
@ 2024-03-28 10:30 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 76f48b13d2..b93000adc4 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Thu_Mar_28_19_59_25_2024_076)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v16 8/8] Allow to print raw parse tree.
@ 2024-04-12 06:49 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-04-12 06:49 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 76f48b13d2..b93000adc4 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Fri_Apr_12_16_09_08_2024_262)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v17 8/8] Allow to print raw parse tree.
@ 2024-04-28 11:00 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-04-28 11:00 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2dff28afce..db6e91f5d6 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Sun_Apr_28_20_28_26_2024_444)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v18 8/8] Allow to print raw parse tree.
@ 2024-05-11 07:11 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-05-11 07:11 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2dff28afce..db6e91f5d6 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Sat_May_11_16_23_07_2024_789)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v19 8/8] Allow to print raw parse tree.
@ 2024-05-14 23:26 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2dff28afce..db6e91f5d6 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Wed_May_15_09_02_03_2024_008)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v20 8/8] Allow to print raw parse tree.
@ 2024-05-24 02:26 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 45a3794b8e..ecbf8e7999 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -653,6 +653,10 @@ pg_parse_query(const char *query_string)
}
#endif
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Fri_May_24_11_39_19_2024_763)----
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v21 8/8] Allow to print raw parse tree.
@ 2024-08-26 04:32 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Tatsuo Ishii @ 2024-08-26 04:32 UTC (permalink / raw)
---
src/backend/tcop/postgres.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8bc6bea113..f15659bf2b 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -659,6 +659,10 @@ pg_parse_query(const char *query_string)
#endif /* DEBUG_NODE_TESTS_ENABLED */
+ if (Debug_print_parse)
+ elog_node_display(LOG, "raw parse tree", raw_parsetree_list,
+ Debug_pretty_print);
+
TRACE_POSTGRESQL_QUERY_PARSE_DONE(query_string);
return raw_parsetree_list;
--
2.25.1
----Next_Part(Mon_Aug_26_13_39_47_2024_878)----
^ permalink raw reply [nested|flat] 51+ messages in thread
end of thread, other threads:[~2024-08-26 04:32 UTC | newest]
Thread overview: 51+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-12-28 04:21 add_partial_path() may remove dominated path but still in use Kohei KaiGai <[email protected]>
2018-12-28 16:44 ` Tom Lane <[email protected]>
2018-12-29 01:05 ` Kohei KaiGai <[email protected]>
2018-12-29 19:12 ` Tom Lane <[email protected]>
2018-12-30 03:31 ` Kohei KaiGai <[email protected]>
2018-12-31 04:09 ` Amit Kapila <[email protected]>
2018-12-31 12:17 ` Kohei KaiGai <[email protected]>
2018-12-31 13:25 ` Amit Kapila <[email protected]>
2019-01-02 13:34 ` Kohei KaiGai <[email protected]>
2019-01-04 04:46 ` Kohei KaiGai <[email protected]>
2019-01-09 04:18 ` Kyotaro HORIGUCHI <[email protected]>
2019-01-09 05:44 ` Kohei KaiGai <[email protected]>
2019-01-10 20:52 ` Robert Haas <[email protected]>
2019-01-11 02:09 ` Kohei KaiGai <[email protected]>
2019-01-11 16:36 ` Robert Haas <[email protected]>
2019-01-17 09:28 ` Kyotaro HORIGUCHI <[email protected]>
2019-01-22 11:50 ` Kohei KaiGai <[email protected]>
2023-05-03 21:59 Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-03 22:48 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 22:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-05-05 02:06 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-12 01:18 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-12 17:36 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
2023-05-12 17:37 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Ryan Booz <[email protected]>
2023-05-13 02:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:04 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-11 20:40 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-11 20:49 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-14 02:46 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Greg Stark <[email protected]>
2023-05-14 20:59 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-15 03:10 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing Peter Geoghegan <[email protected]>
2023-05-04 21:57 ` Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing samay sharma <[email protected]>
2023-07-26 10:49 [PATCH v3 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-08-09 07:56 [PATCH v4 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-09-02 06:32 [PATCH v5 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-09-12 05:22 [PATCH v6 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-09-22 04:53 [PATCH v7 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-09-25 05:01 [PATCH v8 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-10-04 05:51 [PATCH v9 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-10-22 02:22 [PATCH v10 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-11-08 06:57 [PATCH v11 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2023-12-04 11:23 [PATCH v12 7/7] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-01-22 09:45 [PATCH v13 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-02-28 13:59 [PATCH v14 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-03-28 10:30 [PATCH v15 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-04-12 06:49 [PATCH v16 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-04-28 11:00 [PATCH v17 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-05-11 07:11 [PATCH v18 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-05-14 23:26 [PATCH v19 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-05-24 02:26 [PATCH v20 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
2024-08-26 04:32 [PATCH v21 8/8] Allow to print raw parse tree. Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox