public inbox for [email protected]
help / color / mirror / Atom feedRe: Patch: Write Amplification Reduction Method (WARM)
104+ messages / 13 participants
[nested] [flat]
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-08 17:00 Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-08 17:00 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Here's a rebased set of patches. This is the same Pavan posted; I only
fixed some whitespace and a trivial conflict in indexam.c, per 9b88f27cb42f.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-08 17:08 Robert Haas <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-08 17:08 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 8, 2017 at 12:00 PM, Alvaro Herrera
<[email protected]> wrote:
> Here's a rebased set of patches. This is the same Pavan posted; I only
> fixed some whitespace and a trivial conflict in indexam.c, per 9b88f27cb42f.
No attachments.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-08 17:14 Alvaro Herrera <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 3 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-08 17:14 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Alvaro Herrera wrote:
> Here's a rebased set of patches. This is the same Pavan posted; I only
> fixed some whitespace and a trivial conflict in indexam.c, per 9b88f27cb42f.
Jaime noted that I forgot the attachments. Here they are
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-08 18:18 Robert Haas <[email protected]>
parent: Alvaro Herrera <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Robert Haas @ 2017-03-08 18:18 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 8, 2017 at 12:14 PM, Alvaro Herrera
<[email protected]> wrote:
> Alvaro Herrera wrote:
>> Here's a rebased set of patches. This is the same Pavan posted; I only
>> fixed some whitespace and a trivial conflict in indexam.c, per 9b88f27cb42f.
>
> Jaime noted that I forgot the attachments. Here they are
If I recall correctly, the main concern about 0001 was whether it
might negatively affect performance, and testing showed that, if
anything, it was a little better. Does that sound right?
Regarding 0002, I think this could use some documentation someplace
explaining the overall theory of operation. README.HOT, maybe?
+ * Most often and unless we are dealing with a pg-upgraded cluster, the
+ * root offset information should be cached. So there should not be too
+ * much overhead of fetching this information. Also, once a tuple is
+ * updated, the information will be copied to the new version. So it's not
+ * as if we're going to pay this price forever.
What if a tuple is updated -- presumably clearing the
HEAP_LATEST_TUPLE on the tuple at the end of the chain -- and then the
update aborts? Then we must be back to not having this information.
One overall question about this patch series is how we feel about
using up this many bits. 0002 uses a bit from infomask, and 0005 uses
a bit from infomask2. I'm not sure if that's everything, and then I
think we're steeling some bits from the item pointers, too. While the
performance benefits of the patch sound pretty good based on the test
results so far, this is definitely the very last time we'll be able to
implement a feature that requires this many bits.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-08 19:30 Alvaro Herrera <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-08 19:30 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Robert Haas wrote:
> On Wed, Mar 8, 2017 at 12:14 PM, Alvaro Herrera
> <[email protected]> wrote:
> > Alvaro Herrera wrote:
> >> Here's a rebased set of patches. This is the same Pavan posted; I only
> >> fixed some whitespace and a trivial conflict in indexam.c, per 9b88f27cb42f.
> >
> > Jaime noted that I forgot the attachments. Here they are
>
> If I recall correctly, the main concern about 0001 was whether it
> might negatively affect performance, and testing showed that, if
> anything, it was a little better. Does that sound right?
Not really -- it's a bit slower actually in a synthetic case measuring
exactly the slowed-down case. See
https://www.postgresql.org/message-id/[email protected]...
I bet in normal cases it's unnoticeable. If WARM flies, then it's going
to provide a larger improvement than is lost to this.
> Regarding 0002, I think this could use some documentation someplace
> explaining the overall theory of operation. README.HOT, maybe?
Hmm. Yeah, we should have something to that effect. 0005 includes
README.WARM, but I think there should be some place unified that
explains the whole thing.
> + * Most often and unless we are dealing with a pg-upgraded cluster, the
> + * root offset information should be cached. So there should not be too
> + * much overhead of fetching this information. Also, once a tuple is
> + * updated, the information will be copied to the new version. So it's not
> + * as if we're going to pay this price forever.
>
> What if a tuple is updated -- presumably clearing the
> HEAP_LATEST_TUPLE on the tuple at the end of the chain -- and then the
> update aborts? Then we must be back to not having this information.
I will leave this question until I have grokked how this actually works.
> One overall question about this patch series is how we feel about
> using up this many bits. 0002 uses a bit from infomask, and 0005 uses
> a bit from infomask2. I'm not sure if that's everything, and then I
> think we're steeling some bits from the item pointers, too. While the
> performance benefits of the patch sound pretty good based on the test
> results so far, this is definitely the very last time we'll be able to
> implement a feature that requires this many bits.
Yeah, this patch series uses a lot of bits. At some point we should
really add the "last full-scanned by version X" we discussed a long time
ago, and free the MOVED_IN / MOVED_OFF bits that have been unused for so
long. Sadly, once we add that, we need to wait one more release before
we can use the bits anyway.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-09 03:13 Robert Haas <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-09 03:13 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera <[email protected]> wrote:
> Not really -- it's a bit slower actually in a synthetic case measuring
> exactly the slowed-down case. See
> https://www.postgresql.org/message-id/[email protected]...
> I bet in normal cases it's unnoticeable. If WARM flies, then it's going
> to provide a larger improvement than is lost to this.
Hmm, that test case isn't all that synthetic. It's just a single
column bulk update, which isn't anything all that crazy, and 5-10%
isn't nothing.
I'm kinda surprised it made that much difference, though.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-10 18:07 Alvaro Herrera <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-10 18:07 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Robert Haas wrote:
> On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera <[email protected]> wrote:
> > Not really -- it's a bit slower actually in a synthetic case measuring
> > exactly the slowed-down case. See
> > https://www.postgresql.org/message-id/[email protected]...
> > I bet in normal cases it's unnoticeable. If WARM flies, then it's going
> > to provide a larger improvement than is lost to this.
>
> Hmm, that test case isn't all that synthetic. It's just a single
> column bulk update, which isn't anything all that crazy,
The problem is that the update touches the second indexed column. With
the original code we would have stopped checking at that point, but with
the patched code we continue to verify all the other indexed columns for
changes.
Maybe we need more than one bitmapset to be given -- multiple ones for
for "any of these" checks (such as HOT, KEY and Identity) which can be
stopped as soon as one is found, and one for "all of these" (for WARM,
indirect indexes) which needs to be checked to completion.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 01:47 Alvaro Herrera <[email protected]>
parent: Alvaro Herrera <[email protected]>
2 siblings, 2 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-14 01:47 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
> @@ -234,6 +236,21 @@ index_beginscan(Relation heapRelation,
> scan->heapRelation = heapRelation;
> scan->xs_snapshot = snapshot;
>
> + /*
> + * If the index supports recheck, make sure that index tuple is saved
> + * during index scans.
> + *
> + * XXX Ideally, we should look at all indexes on the table and check if
> + * WARM is at all supported on the base table. If WARM is not supported
> + * then we don't need to do any recheck. RelationGetIndexAttrBitmap() does
> + * do that and sets rd_supportswarm after looking at all indexes. But we
> + * don't know if the function was called earlier in the session when we're
> + * here. We can't call it now because there exists a risk of causing
> + * deadlock.
> + */
> + if (indexRelation->rd_amroutine->amrecheck)
> + scan->xs_want_itup = true;
> +
> return scan;
> }
I didn't like this comment very much. But it's not necessary: you have
already given relcache responsibility for setting rd_supportswarm. The
only problem seems to be that you set it in RelationGetIndexAttrBitmap
instead of RelationGetIndexList, but it's not clear to me why. I think
if the latter function is in charge, then we can trust the flag more
than the current situation. Let's set the value to false on relcache
entry build, for safety's sake.
I noticed that nbtinsert.c and nbtree.c have a bunch of new includes
that they don't actually need. Let's remove those. nbtutils.c does
need them because of btrecheck(). Speaking of which:
I have already commented about the executor involvement in btrecheck();
that doesn't seem good. I previously suggested to pass the EState down
from caller, but that's not a great idea either since you still need to
do the actual FormIndexDatum. I now think that a workable option would
be to compute the values/isnulls arrays so that btrecheck gets them
already computed. With that, this function would be no more of a
modularity violation that HeapSatisfiesHOTAndKey() itself.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 17:17 Alvaro Herrera <[email protected]>
parent: Alvaro Herrera <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-14 17:17 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
After looking at how index_fetch_heap and heap_hot_search_buffer
interact, I can't say I'm in love with the idea. I started thinking
that we should not have index_fetch_heap release the buffer lock only to
re-acquire it five lines later, so it should keep the buffer lock, do
the recheck and only release it afterwards (I realize that this means
there'd be need for two additional "else release buffer lock" branches);
but then this got me thinking that perhaps it would be better to have
another routine that does both call heap_hot_search_buffer and then call
recheck -- it occurs to me that what we're doing here is essentially
heap_warm_search_buffer.
Does that make sense?
Another thing is BuildIndexInfo being called over and over for each
recheck(). Surely we need to cache the indexinfo for each indexscan.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 18:41 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-14 18:41 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <[email protected]>
wrote:
> > @@ -234,6 +236,21 @@ index_beginscan(Relation heapRelation,
> > scan->heapRelation = heapRelation;
> > scan->xs_snapshot = snapshot;
> >
> > + /*
> > + * If the index supports recheck, make sure that index tuple is
> saved
> > + * during index scans.
> > + *
> > + * XXX Ideally, we should look at all indexes on the table and
> check if
> > + * WARM is at all supported on the base table. If WARM is not
> supported
> > + * then we don't need to do any recheck.
> RelationGetIndexAttrBitmap() does
> > + * do that and sets rd_supportswarm after looking at all indexes.
> But we
> > + * don't know if the function was called earlier in the session
> when we're
> > + * here. We can't call it now because there exists a risk of
> causing
> > + * deadlock.
> > + */
> > + if (indexRelation->rd_amroutine->amrecheck)
> > + scan->xs_want_itup = true;
> > +
> > return scan;
> > }
>
> I didn't like this comment very much. But it's not necessary: you have
> already given relcache responsibility for setting rd_supportswarm. The
> only problem seems to be that you set it in RelationGetIndexAttrBitmap
> instead of RelationGetIndexList, but it's not clear to me why.
Hmm. I think you're right. Will fix that way and test.
>
> I noticed that nbtinsert.c and nbtree.c have a bunch of new includes
> that they don't actually need. Let's remove those. nbtutils.c does
> need them because of btrecheck().
Right. It's probably a left over from the way I wrote the first version.
Will fix.
Speaking of which:
>
> I have already commented about the executor involvement in btrecheck();
> that doesn't seem good. I previously suggested to pass the EState down
> from caller, but that's not a great idea either since you still need to
> do the actual FormIndexDatum. I now think that a workable option would
> be to compute the values/isnulls arrays so that btrecheck gets them
> already computed.
I agree with your complaint about modularity violation. What I am unclear
is how passing values/isnulls array will fix that. The way code is
structured currently, recheck routines are called by index_fetch_heap(). So
if we try to compute values/isnulls in that function, we'll still need
access EState, which AFAIU will lead to similar violation. Or am I
mis-reading your idea?
I wonder if we should instead invent something similar to IndexRecheck(),
but instead of running ExecQual(), this new routine will compare the index
values by the given HeapTuple against given IndexTuple. ISTM that for this
to work we'll need to modify all callers of index_getnext() and teach them
to invoke the AM specific recheck method if xs_tuple_recheck flag is set to
true by index_getnext().
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 19:15 Pavan Deolasee <[email protected]>
parent: Pavan Deolasee <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-14 19:15 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 5:17 PM, Alvaro Herrera <[email protected]>
wrote:
> After looking at how index_fetch_heap and heap_hot_search_buffer
> interact, I can't say I'm in love with the idea. I started thinking
> that we should not have index_fetch_heap release the buffer lock only to
> re-acquire it five lines later, so it should keep the buffer lock, do
> the recheck and only release it afterwards (I realize that this means
> there'd be need for two additional "else release buffer lock" branches);
>
Yes, it makes sense.
> but then this got me thinking that perhaps it would be better to have
> another routine that does both call heap_hot_search_buffer and then call
> recheck -- it occurs to me that what we're doing here is essentially
> heap_warm_search_buffer.
>
> Does that make sense?
>
We can do that, but it's not clear to me if that would be a huge
improvement. Also, I think we need to first decide on how to model the
recheck logic since that might affect this function significantly. For
example, if we decide to do recheck at a higher level then we will most
likely end up releasing and reacquiring the lock anyways.
>
> Another thing is BuildIndexInfo being called over and over for each
> recheck(). Surely we need to cache the indexinfo for each indexscan.
>
>
Good point. What should that place be though? Can we just cache them in the
relcache and maintain them along with the list of indexes? Looking at the
current callers, ExecOpenIndices() usually cache them in the ResultRelInfo,
which is sufficient because INSERT/UPDATE/DELETE code paths are the most
relevant paths where caching definitely helps. The only other place where
it may get called once per tuple is unique_key_recheck(), which is used for
deferred unique key tests and hence probably not very common.
BTW I wanted to share some more numbers from a recent performance test. I
thought it's important because the latest patch has fully functional chain
conversion code as well as all WAL-logging related pieces are in place
too. I ran these tests on a box borrowed from Tomas (thanks!). This has
64GB RAM and 350GB SSD with 1GB on-board RAM. I used the same test setup
that I used for the first test results reported on this thread i.e. a
modified pgbench_accounts table with additional columns and additional
indexes (one index on abalance so that every UPDATE is a potential WARM
update).
In a test where table + indexes exceeds RAM, running for 8hrs and
auto-vacuum parameters set such that we get 2-3 autovacuums on the table
during the test, we see WARM delivering more than 100% TPS as compared to
master. In this graph, I've plotted a moving average of TPS and the spikes
that we see coincides with the checkpoints (checkpoint_timeout is set to
20mins and max_wal_size large enough to avoid any xlog-based checkpoints).
The spikes are more prominent on WARM but I guess that's purely because it
delivers much higher TPS. I haven't shown here but I see WARM updates close
to 65-70% of the total updates. Also there is significant reduction in WAL
generated per txn.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/pdf] Moderate_AV_4Indexes_100FF_SF1200_Duration28800s_Run2.pdf (225.5K, ../../CABOikdN-G95zpCiJS2FXbbQRYT3M=qdAWasTx3kwOsqKWE=5uQ@mail.gmail.com/3-Moderate_AV_4Indexes_100FF_SF1200_Duration28800s_Run2.pdf)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 19:16 Alvaro Herrera <[email protected]>
parent: Pavan Deolasee <[email protected]>
1 sibling, 2 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-14 19:16 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Pavan Deolasee wrote:
> On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <[email protected]>
> wrote:
> > I have already commented about the executor involvement in btrecheck();
> > that doesn't seem good. I previously suggested to pass the EState down
> > from caller, but that's not a great idea either since you still need to
> > do the actual FormIndexDatum. I now think that a workable option would
> > be to compute the values/isnulls arrays so that btrecheck gets them
> > already computed.
>
> I agree with your complaint about modularity violation. What I am unclear
> is how passing values/isnulls array will fix that. The way code is
> structured currently, recheck routines are called by index_fetch_heap(). So
> if we try to compute values/isnulls in that function, we'll still need
> access EState, which AFAIU will lead to similar violation. Or am I
> mis-reading your idea?
You're right, it's still a problem. (Honestly, I think the whole idea
of trying to compute a fake index tuple starting from a just-read heap
tuple is a problem in itself; I just wonder if there's a way to do the
recheck that doesn't involve such a thing.)
> I wonder if we should instead invent something similar to IndexRecheck(),
> but instead of running ExecQual(), this new routine will compare the index
> values by the given HeapTuple against given IndexTuple. ISTM that for this
> to work we'll need to modify all callers of index_getnext() and teach them
> to invoke the AM specific recheck method if xs_tuple_recheck flag is set to
> true by index_getnext().
Yeah, grumble, that idea does sound intrusive, but perhaps it's
workable. What about bitmap indexscans? AFAICS we already have a
recheck there natively, so we only need to mark the page as lossy, which
we're already doing anyway.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 19:19 Alvaro Herrera <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-14 19:19 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Pavan Deolasee wrote:
> BTW I wanted to share some more numbers from a recent performance test. I
> thought it's important because the latest patch has fully functional chain
> conversion code as well as all WAL-logging related pieces are in place
> too. I ran these tests on a box borrowed from Tomas (thanks!). This has
> 64GB RAM and 350GB SSD with 1GB on-board RAM. I used the same test setup
> that I used for the first test results reported on this thread i.e. a
> modified pgbench_accounts table with additional columns and additional
> indexes (one index on abalance so that every UPDATE is a potential WARM
> update).
>
> In a test where table + indexes exceeds RAM, running for 8hrs and
> auto-vacuum parameters set such that we get 2-3 autovacuums on the table
> during the test, we see WARM delivering more than 100% TPS as compared to
> master. In this graph, I've plotted a moving average of TPS and the spikes
> that we see coincides with the checkpoints (checkpoint_timeout is set to
> 20mins and max_wal_size large enough to avoid any xlog-based checkpoints).
> The spikes are more prominent on WARM but I guess that's purely because it
> delivers much higher TPS. I haven't shown here but I see WARM updates close
> to 65-70% of the total updates. Also there is significant reduction in WAL
> generated per txn.
Impressive results. Labels on axes would improve readability of the chart :-)
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 19:24 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-14 19:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 7:19 PM, Alvaro Herrera <[email protected]>
wrote:
> Pavan Deolasee wrote:
>
> > BTW I wanted to share some more numbers from a recent performance test. I
> > thought it's important because the latest patch has fully functional
> chain
> > conversion code as well as all WAL-logging related pieces are in place
> > too. I ran these tests on a box borrowed from Tomas (thanks!). This has
> > 64GB RAM and 350GB SSD with 1GB on-board RAM. I used the same test setup
> > that I used for the first test results reported on this thread i.e. a
> > modified pgbench_accounts table with additional columns and additional
> > indexes (one index on abalance so that every UPDATE is a potential WARM
> > update).
> >
> > In a test where table + indexes exceeds RAM, running for 8hrs and
> > auto-vacuum parameters set such that we get 2-3 autovacuums on the table
> > during the test, we see WARM delivering more than 100% TPS as compared to
> > master. In this graph, I've plotted a moving average of TPS and the
> spikes
> > that we see coincides with the checkpoints (checkpoint_timeout is set to
> > 20mins and max_wal_size large enough to avoid any xlog-based
> checkpoints).
> > The spikes are more prominent on WARM but I guess that's purely because
> it
> > delivers much higher TPS. I haven't shown here but I see WARM updates
> close
> > to 65-70% of the total updates. Also there is significant reduction in
> WAL
> > generated per txn.
>
> Impressive results. Labels on axes would improve readability of the chart
> :-)
>
>
Sorry about that. I was desperately searching for Undo button after hitting
"send" for the very same reason :-) Looks like I used gnuplot after a few
years.
Just to make it clear, the X-axis is duration of tests in seconds and
Y-axis is 450s moving average of TPS. BTW 450 is no magic figure. I
collected stats every 15s and took a moving average of last 30 samples.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-14 20:14 Peter Geoghegan <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Peter Geoghegan @ 2017-03-14 20:14 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 12:19 PM, Alvaro Herrera
<[email protected]> wrote:
> Impressive results.
Agreed.
It seems like an important invariant for WARM is that any duplicate
index values ought to have different TIDs (actually, it's a bit
stricter than that, since btrecheck() cares about simple binary
equality). ISTM that it would be fairly easy to modify amcheck such
that the "items in logical order" check, as well as the similar
"cross-page order" check (the one that detects transposed pages) also
check that this new WARM invariant holds. Obviously this would only
make sense on the leaf level of the index.
You wouldn't have to teach amcheck about the heap, because a TID that
points to the heap can only be duplicated within a B-Tree index
because of WARM. So, if we find that two adjacent tuples are equal,
check if the TIDs are equal. If they are also equal, check for strict
binary equality. If strict binary equality is indicated, throw an
error due to invariant failing.
IIUC, the design of WARM makes this simple enough to implement, and
cheap enough that the additional runtime overhead is well worthwhile.
You could just add this check to the existing checks without changing
the user-visible interface. It seems pretty complementary to what is
already there.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-15 19:44 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-15 19:44 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 7:16 PM, Alvaro Herrera <[email protected]>
wrote:
> Pavan Deolasee wrote:
> > On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <
> [email protected]>
> > wrote:
>
> > > I have already commented about the executor involvement in btrecheck();
> > > that doesn't seem good. I previously suggested to pass the EState down
> > > from caller, but that's not a great idea either since you still need to
> > > do the actual FormIndexDatum. I now think that a workable option would
> > > be to compute the values/isnulls arrays so that btrecheck gets them
> > > already computed.
> >
> > I agree with your complaint about modularity violation. What I am unclear
> > is how passing values/isnulls array will fix that. The way code is
> > structured currently, recheck routines are called by index_fetch_heap().
> So
> > if we try to compute values/isnulls in that function, we'll still need
> > access EState, which AFAIU will lead to similar violation. Or am I
> > mis-reading your idea?
>
> You're right, it's still a problem. (Honestly, I think the whole idea
> of trying to compute a fake index tuple starting from a just-read heap
> tuple is a problem in itself;
Why do you think so?
> I just wonder if there's a way to do the
> recheck that doesn't involve such a thing.)
>
I couldn't find a better way without a lot of complex infrastructure. Even
though we now have ability to mark index pointers and we know that a given
pointer either points to the pre-WARM chain or post-WARM chain, this does
not solve the case when an index does not receive a new entry. In that
case, both pre-WARM and post-WARM tuples are reachable via the same old
index pointer. The only way we could deal with this is to mark index
pointers as "common", "pre-warm" and "post-warm". But that would require us
to update the old pointer's state from "common" to "pre-warm" for the index
whose keys are being updated. May be it's doable, but might be more complex
than the current approach.
>
> > I wonder if we should instead invent something similar to IndexRecheck(),
> > but instead of running ExecQual(), this new routine will compare the
> index
> > values by the given HeapTuple against given IndexTuple. ISTM that for
> this
> > to work we'll need to modify all callers of index_getnext() and teach
> them
> > to invoke the AM specific recheck method if xs_tuple_recheck flag is set
> to
> > true by index_getnext().
>
> Yeah, grumble, that idea does sound intrusive, but perhaps it's
> workable. What about bitmap indexscans? AFAICS we already have a
> recheck there natively, so we only need to mark the page as lossy, which
> we're already doing anyway.
>
Yeah, bitmap indexscans should be ok. We need recheck logic only to avoid
duplicate scans and since a TID can only occur once in the bitmap, there is
no risk for duplicate results.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-16 12:53 Robert Haas <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Robert Haas @ 2017-03-16 12:53 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 15, 2017 at 3:44 PM, Pavan Deolasee
<[email protected]> wrote:
> I couldn't find a better way without a lot of complex infrastructure. Even
> though we now have ability to mark index pointers and we know that a given
> pointer either points to the pre-WARM chain or post-WARM chain, this does
> not solve the case when an index does not receive a new entry. In that case,
> both pre-WARM and post-WARM tuples are reachable via the same old index
> pointer. The only way we could deal with this is to mark index pointers as
> "common", "pre-warm" and "post-warm". But that would require us to update
> the old pointer's state from "common" to "pre-warm" for the index whose keys
> are being updated. May be it's doable, but might be more complex than the
> current approach.
/me scratches head.
Aren't pre-warm and post-warm just (better) names for blue and red?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-19 07:05 Pavan Deolasee <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-19 07:05 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 16, 2017 at 12:53 PM, Robert Haas <[email protected]> wrote:
> On Wed, Mar 15, 2017 at 3:44 PM, Pavan Deolasee
> <[email protected]> wrote:
> > I couldn't find a better way without a lot of complex infrastructure.
> Even
> > though we now have ability to mark index pointers and we know that a
> given
> > pointer either points to the pre-WARM chain or post-WARM chain, this does
> > not solve the case when an index does not receive a new entry. In that
> case,
> > both pre-WARM and post-WARM tuples are reachable via the same old index
> > pointer. The only way we could deal with this is to mark index pointers
> as
> > "common", "pre-warm" and "post-warm". But that would require us to update
> > the old pointer's state from "common" to "pre-warm" for the index whose
> keys
> > are being updated. May be it's doable, but might be more complex than the
> > current approach.
>
> /me scratches head.
>
> Aren't pre-warm and post-warm just (better) names for blue and red?
>
>
Yeah, sounds better. Just to make it clear, the current design sets the
following information:
HEAP_WARM_TUPLE - When a row gets WARM updated, both old and new versions
of the row are marked with HEAP_WARM_TUPLE flag. This allows us to remember
that a certain row was WARM-updated, even if the update later aborts and we
cleanup the new version and truncate the chain. All subsequent tuple
versions will carry this flag until a non-HOT updates happens, which breaks
the HOT chain.
HEAP_WARM_RED - After first WARM update, the new version of the tuple is
marked with this flag. This flag is also carried forward to all future HOT
updated tuples. So the only tuple that has HEAP_WARM_TUPLE but not
HEAP_WARM_RED is the old version before the WARM update. Also, all tuples
marked with HEAP_WARM_RED flag satisfies HOT property (i.e. all index key
columns share the same value). Similarly, all tuples NOT marked with
HEAP_WARM_RED also satisfy HOT property. I've so far called them Red and
Blue chains respectively.
In addition, in the current patch, the new index pointers resulted from
WARM updates are marked BTREE_INDEX_RED_POINTER/HASH_INDEX_RED_POINTER
I think per your suggestion we can change HEAP_WARM_RED to HEAP_WARM_TUPLE
and similarly rename the index pointers to BTREE/HASH_INDEX_WARM_POINTER
and replace HEAP_WARM_TUPLE with something like HEAP_WARM_UPDATED_TUPLE to
signify that this or some previous version of this chain was once
WARM-updated.
Does that sound ok? I can change the patch accordingly.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-19 07:15 Pavan Deolasee <[email protected]>
parent: Peter Geoghegan <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-19 07:15 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 8:14 PM, Peter Geoghegan <[email protected]> wrote:
> On Tue, Mar 14, 2017 at 12:19 PM, Alvaro Herrera
> <[email protected]> wrote:
> > Impressive results.
>
> Agreed.
>
Thanks. I repeated the same tests with slightly lower scale factor so that
most (but not all) data fits in memory. The results are kinda similar
(attached here). The spikes are still there and they correspond to the
checkpoint_timeout set for these tests.
> It seems like an important invariant for WARM is that any duplicate
> index values ought to have different TIDs (actually, it's a bit
> stricter than that, since btrecheck() cares about simple binary
> equality).
Yes. I think in the current code, indexes can never duplicate TIDs (at
least for btrees and hash). With WARM, indexes can have duplicate TIDs, but
iff index values differ. In addition there can only be one more duplicate
and one of them must be a Blue pointer (or a non-WARM pointer if we accept
the new nomenclature proposed a few mins back).
>
> You wouldn't have to teach amcheck about the heap, because a TID that
> points to the heap can only be duplicated within a B-Tree index
> because of WARM. So, if we find that two adjacent tuples are equal,
> check if the TIDs are equal. If they are also equal, check for strict
> binary equality. If strict binary equality is indicated, throw an
> error due to invariant failing.
>
>
Wouldn't this be much more expensive for non-unique indexes?
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/pdf] Moderate_AV_4Indexes_100FF_SF800_Duration28800s.pdf (248.7K, ../../CABOikdNV253GMj3CwQEjrHiYWVJ-m9uLO8SqSSxBDJQKw9RMeA@mail.gmail.com/3-Moderate_AV_4Indexes_100FF_SF800_Duration28800s.pdf)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 14:41 Robert Haas <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Robert Haas @ 2017-03-20 14:41 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Sun, Mar 19, 2017 at 3:05 AM, Pavan Deolasee
<[email protected]> wrote:
> On Thu, Mar 16, 2017 at 12:53 PM, Robert Haas <[email protected]> wrote:
>> On Wed, Mar 15, 2017 at 3:44 PM, Pavan Deolasee
>> <[email protected]> wrote:
>> > I couldn't find a better way without a lot of complex infrastructure.
>> > Even
>> > though we now have ability to mark index pointers and we know that a
>> > given
>> > pointer either points to the pre-WARM chain or post-WARM chain, this
>> > does
>> > not solve the case when an index does not receive a new entry. In that
>> > case,
>> > both pre-WARM and post-WARM tuples are reachable via the same old index
>> > pointer. The only way we could deal with this is to mark index pointers
>> > as
>> > "common", "pre-warm" and "post-warm". But that would require us to
>> > update
>> > the old pointer's state from "common" to "pre-warm" for the index whose
>> > keys
>> > are being updated. May be it's doable, but might be more complex than
>> > the
>> > current approach.
>>
>> /me scratches head.
>>
>> Aren't pre-warm and post-warm just (better) names for blue and red?
>>
>
> Yeah, sounds better.
My point here wasn't really about renaming, although I do think
renaming is something that should get done. My point was that you
were saying we need to mark index pointers as common, pre-warm, and
post-warm. But you're pretty much already doing that, I think. I
guess you don't have "common", but you do have "pre-warm" and
"post-warm".
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 15:00 Pavan Deolasee <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-20 15:00 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Mon, Mar 20, 2017 at 8:11 PM, Robert Haas <[email protected]> wrote:
> On Sun, Mar 19, 2017 at 3:05 AM, Pavan Deolasee
> <[email protected]> wrote:
> > On Thu, Mar 16, 2017 at 12:53 PM, Robert Haas <[email protected]>
> wrote:
>
> >>
> >> /me scratches head.
> >>
> >> Aren't pre-warm and post-warm just (better) names for blue and red?
> >>
> >
> > Yeah, sounds better.
>
> My point here wasn't really about renaming, although I do think
> renaming is something that should get done. My point was that you
> were saying we need to mark index pointers as common, pre-warm, and
> post-warm. But you're pretty much already doing that, I think. I
> guess you don't have "common", but you do have "pre-warm" and
> "post-warm".
>
>
Ah, I mis-read that. Strictly speaking, we already have common (blue) and
post-warm (red), and I just finished renaming them to CLEAR (of WARM bit)
and WARM. May be it's still not the best name, but I think it looks better
than before.
But the larger point is that we don't have an easy to know if an index
pointer which was inserted with the original heap tuple (i.e. pre-WARM
update) should only return pre-WARM tuples or should it also return
post-WARM tuples. Right now we make that decision by looking at the
index-keys and discard the pointer whose index-key does not match the ones
created from heap-keys. If we need to change that then at every WARM
update, we will have to go back to the original pointer and change it's
state to pre-warm. That looks more invasive and requires additional index
management.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 15:03 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-20 15:03 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 15, 2017 at 12:46 AM, Alvaro Herrera <[email protected]>
wrote:
> Pavan Deolasee wrote:
> > On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <
> [email protected]>
> > wrote:
>
> > > I have already commented about the executor involvement in btrecheck();
> > > that doesn't seem good. I previously suggested to pass the EState down
> > > from caller, but that's not a great idea either since you still need to
> > > do the actual FormIndexDatum. I now think that a workable option would
> > > be to compute the values/isnulls arrays so that btrecheck gets them
> > > already computed.
> >
> > I agree with your complaint about modularity violation. What I am unclear
> > is how passing values/isnulls array will fix that. The way code is
> > structured currently, recheck routines are called by index_fetch_heap().
> So
> > if we try to compute values/isnulls in that function, we'll still need
> > access EState, which AFAIU will lead to similar violation. Or am I
> > mis-reading your idea?
>
> You're right, it's still a problem.
BTW I realised that we don't really need those executor bits in recheck
routines. We don't support WARM when attributes in index expressions are
modified. So we really don't need to do any comparison for those
attributes. I've written a separate form of FormIndexDatum() which will
only return basic index attributes and comparing them should be enough.
Will share rebased and updated patch soon.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 15:09 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-20 15:09 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <[email protected]>
wrote:
> > @@ -234,6 +236,21 @@ index_beginscan(Relation heapRelation,
> > scan->heapRelation = heapRelation;
> > scan->xs_snapshot = snapshot;
> >
> > + /*
> > + * If the index supports recheck, make sure that index tuple is
> saved
> > + * during index scans.
> > + *
> > + * XXX Ideally, we should look at all indexes on the table and
> check if
> > + * WARM is at all supported on the base table. If WARM is not
> supported
> > + * then we don't need to do any recheck.
> RelationGetIndexAttrBitmap() does
> > + * do that and sets rd_supportswarm after looking at all indexes.
> But we
> > + * don't know if the function was called earlier in the session
> when we're
> > + * here. We can't call it now because there exists a risk of
> causing
> > + * deadlock.
> > + */
> > + if (indexRelation->rd_amroutine->amrecheck)
> > + scan->xs_want_itup = true;
> > +
> > return scan;
> > }
>
> I didn't like this comment very much. But it's not necessary: you have
> already given relcache responsibility for setting rd_supportswarm. The
> only problem seems to be that you set it in RelationGetIndexAttrBitmap
> instead of RelationGetIndexList, but it's not clear to me why. I think
> if the latter function is in charge, then we can trust the flag more
> than the current situation.
I looked at this today. AFAICS we don't have access to rd_amroutine in
RelationGetIndexList since we don't actually call index_open() in that
function. Would it be safe to do that? I'll give it a shot, but thought of
asking here first.
Thanks,
Pavan
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 16:12 Alvaro Herrera <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-20 16:12 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Pavan Deolasee wrote:
> On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera <[email protected]>
> wrote:
> > I didn't like this comment very much. But it's not necessary: you have
> > already given relcache responsibility for setting rd_supportswarm. The
> > only problem seems to be that you set it in RelationGetIndexAttrBitmap
> > instead of RelationGetIndexList, but it's not clear to me why. I think
> > if the latter function is in charge, then we can trust the flag more
> > than the current situation.
>
> I looked at this today. AFAICS we don't have access to rd_amroutine in
> RelationGetIndexList since we don't actually call index_open() in that
> function. Would it be safe to do that? I'll give it a shot, but thought of
> asking here first.
Ah, you're right, we only have the pg_index tuple for the index, not the
pg_am one. I think one pg_am cache lookup isn't really all that
terrible (though we should ensure that there's no circularity problem in
doing that), but I doubt that going to the trouble of invoking the
amhandler just to figure out if it supports WARM is acceptable.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-20 16:46 Peter Geoghegan <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Peter Geoghegan @ 2017-03-20 16:46 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Sun, Mar 19, 2017 at 12:15 AM, Pavan Deolasee
<[email protected]> wrote:
>> It seems like an important invariant for WARM is that any duplicate
>> index values ought to have different TIDs (actually, it's a bit
>> stricter than that, since btrecheck() cares about simple binary
>> equality).
>
> Yes. I think in the current code, indexes can never duplicate TIDs (at least
> for btrees and hash). With WARM, indexes can have duplicate TIDs, but iff
> index values differ. In addition there can only be one more duplicate and
> one of them must be a Blue pointer (or a non-WARM pointer if we accept the
> new nomenclature proposed a few mins back).
It looks like those additional Red/Blue details are available right
from the IndexTuple, which makes the check a good fit for amcheck (no
need to bring the heap into it).
>> You wouldn't have to teach amcheck about the heap, because a TID that
>> points to the heap can only be duplicated within a B-Tree index
>> because of WARM. So, if we find that two adjacent tuples are equal,
>> check if the TIDs are equal. If they are also equal, check for strict
>> binary equality. If strict binary equality is indicated, throw an
>> error due to invariant failing.
>>
>
> Wouldn't this be much more expensive for non-unique indexes?
Only in the worst case, where there are many many duplicates, and only
if you insisted on being completely comprehensive, rather than merely
very comprehensive. That is, you can store the duplicate TIDs in local
memory up to a quasi-arbitrary budget, since you do have to make sure
that any local buffer cannot grow in an unbounded fashion. Certainly,
if you stored 10,000 TIDs, there is always going to be a theoretical
case where that wasn't enough. But you can always say something like
that. We are defending against Murphy here, not Machiavelli.
You're going to have to qsort() a particular value's duplicate TIDs
once you encounter a distinct value, and therefore need to evaluate
the invariant. That's not a big deal, because sorting less than 1,000
items is generally very fast. It's well worth it. I'd probably choose
a generic budget for storing TIDs in local memory, and throw out half
of the TIDs when that budget is exceeded.
I see no difficulty with race conditions when you have only an
AccessShareLock on target. Concurrent page splits won't hurt, because
you reliably skip over those by always moving right. I'm pretty sure
that VACUUM killing IndexTuples that you've already stored with the
intention of sorting later is also not a complicating factor, since
you know that the heap TIDs that are WARM root pointers are not going
to be recycled in the lifetime of the amcheck query such that you get
a false positive.
A WARM check seems like a neat adjunct to what amcheck does already.
It seems like a really good idea for WARM to buy into this kind of
verification. It is, at worst, cheap insurance.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 10:51 Amit Kapila <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Amit Kapila @ 2017-03-21 10:51 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Fri, Mar 10, 2017 at 11:37 PM, Alvaro Herrera
<[email protected]> wrote:
> Robert Haas wrote:
>> On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera <[email protected]> wrote:
>> > Not really -- it's a bit slower actually in a synthetic case measuring
>> > exactly the slowed-down case. See
>> > https://www.postgresql.org/message-id/[email protected]...
>> > I bet in normal cases it's unnoticeable. If WARM flies, then it's going
>> > to provide a larger improvement than is lost to this.
>>
>> Hmm, that test case isn't all that synthetic. It's just a single
>> column bulk update, which isn't anything all that crazy,
>
> The problem is that the update touches the second indexed column. With
> the original code we would have stopped checking at that point, but with
> the patched code we continue to verify all the other indexed columns for
> changes.
>
> Maybe we need more than one bitmapset to be given -- multiple ones for
> for "any of these" checks (such as HOT, KEY and Identity) which can be
> stopped as soon as one is found, and one for "all of these" (for WARM,
> indirect indexes) which needs to be checked to completion.
>
How will that help to mitigate the regression? I think what might
help here is if we fetch the required columns for WARM only when we
know hot_update is false.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 10:56 Amit Kapila <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 2 replies; 104+ messages in thread
From: Amit Kapila @ 2017-03-21 10:56 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 9, 2017 at 8:43 AM, Robert Haas <[email protected]> wrote:
> On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera <[email protected]> wrote:
>> Not really -- it's a bit slower actually in a synthetic case measuring
>> exactly the slowed-down case. See
>> https://www.postgresql.org/message-id/[email protected]...
>> I bet in normal cases it's unnoticeable. If WARM flies, then it's going
>> to provide a larger improvement than is lost to this.
>
> Hmm, that test case isn't all that synthetic. It's just a single
> column bulk update, which isn't anything all that crazy, and 5-10%
> isn't nothing.
>
> I'm kinda surprised it made that much difference, though.
>
I think it is because heap_getattr() is not that cheap. We have
noticed the similar problem during development of scan key push down
work [1].
[1] - https://commitfest.postgresql.org/12/850/
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 12:04 Robert Haas <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 2 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-21 12:04 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila <[email protected]> wrote:
>> Hmm, that test case isn't all that synthetic. It's just a single
>> column bulk update, which isn't anything all that crazy, and 5-10%
>> isn't nothing.
>>
>> I'm kinda surprised it made that much difference, though.
>>
>
> I think it is because heap_getattr() is not that cheap. We have
> noticed the similar problem during development of scan key push down
> work [1].
Yeah. So what's the deal with this? Is somebody working on figuring
out a different approach that would reduce this overhead? Are we
going to defer WARM to v11? Or is the intent to just ignore the 5-10%
slowdown on a single column update and commit everything anyway? (A
strong -1 on that course of action from me.)
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 12:41 Pavan Deolasee <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-21 12:41 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 5:34 PM, Robert Haas <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila <[email protected]>
> wrote:
> >> Hmm, that test case isn't all that synthetic. It's just a single
> >> column bulk update, which isn't anything all that crazy, and 5-10%
> >> isn't nothing.
> >>
> >> I'm kinda surprised it made that much difference, though.
> >>
> >
> > I think it is because heap_getattr() is not that cheap. We have
> > noticed the similar problem during development of scan key push down
> > work [1].
>
> Yeah. So what's the deal with this? Is somebody working on figuring
> out a different approach that would reduce this overhead? Are we
> going to defer WARM to v11? Or is the intent to just ignore the 5-10%
> slowdown on a single column update and commit everything anyway?
I think I should clarify something. The test case does a single column
update, but it also has columns which are very wide, has an index on many
columns (and it updates a column early in the list). In addition, in the
test Mithun updated all 10million rows of the table in a single
transaction, used UNLOGGED table and fsync was turned off.
TBH I see many artificial scenarios here. It will be very useful if he can
rerun the query with some of these restrictions lifted. I'm all for
addressing whatever we can, but I am not sure if this test demonstrates a
real world usage.
Having said that, may be if we can do a few things to reduce the overhead.
- Check if the page has enough free space to perform a HOT/WARM update. If
not, don't look for all index keys.
- Pass bitmaps separately for each index and bail out early if we conclude
neither HOT nor WARM is possible. In this case since there is just one
index and as soon as we check the second column we know neither HOT nor
WARM is possible, we will return early. It might complicate the API a lot,
but I can give it a shot if that's what is needed to make progress.
Any other ideas?
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 13:17 Pavan Deolasee <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-21 13:17 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2017 at 10:47 PM, Alvaro Herrera <[email protected]>
wrote:
> After looking at how index_fetch_heap and heap_hot_search_buffer
> interact, I can't say I'm in love with the idea. I started thinking
> that we should not have index_fetch_heap release the buffer lock only to
> re-acquire it five lines later, so it should keep the buffer lock, do
> the recheck and only release it afterwards (I realize that this means
> there'd be need for two additional "else release buffer lock" branches);
> but then this got me thinking that perhaps it would be better to have
> another routine that does both call heap_hot_search_buffer and then call
> recheck -- it occurs to me that what we're doing here is essentially
> heap_warm_search_buffer.
>
> Does that make sense?
>
> Another thing is BuildIndexInfo being called over and over for each
> recheck(). Surely we need to cache the indexinfo for each indexscan.
>
>
Please find attached rebased patches. There are a few changes in this
version, so let me mention them here instead of trying to reply in-line to
various points on various emails:
1. The patch now has support for hash redo recovery since that was added to
the master (it might be broken since a bug was reported in the original
code itself)
2. Based on Robert's comments and my discussion with him in person, I
removed the Blue/Red naming and instead now using CLEAR and WARM to
identify the parts of the chain and the index pointers. This also resulted
in changes to the way heap tuple header bits are named. So
HEAP_WARM_UPDATED is now used to mark the old tuple which gets WARM updated
and the same flag is copied to all subsequent versions of the tuple, until
a non-HOT updates happens. The new version and all subsequent versions are
marked with HEAP_WARM_TUPLE flag (in the earlier versions this was used for
marking old and the new versions. This might cause confusion, but looks a
more accurate naming to me.
3. IndexInfo is now cached inside IndexScanDescData, which should address
your comment above.
4. I realised that we don't really need to ever compare expression
attributes in the index since WARM is never used when one of those columns
is updated. Hence I've now created a new version of FormIndexDatum which
only returns plain attributes and hence recheck routine does not need
access to any executor stuff.
5. We don't release the lock of the buffer if we are going to apply
recheck. This should address part of the your comment. I haven't though put
them inside a single wrapper function because there is just one caller to
amrecheck function and after this change, it looked ok. But if you don't
still like, I'll make that change.
6. Unnecessary header files included at various places have been removed.
7. Some comments have been updated and rewritten. Hopefully they look
better than before now.
8. I merged the main WARM patch and the chain conversion code in a single
patch since I don't think we will apply them separately. But if it helps
with review, let me know and I can split that again.
9. I realised that we don't really need xs_tuple_recheck in the scan
descriptor and hence removed that and used a stack variable to get that
info.
10. Accidentally WARM was disabled on the system relations during one of
the earlier rebases. So restored that back and made a slight change to
regression expected output.
All tests pass with the patch set. I am now writing TAP tests for WARM and
will submit that separately. Per your suggestion, I am first turning the
stress tests I'd used earlier to use TAP tests and then add more tests,
especially around recovery and index addition/deletion.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0004_freeup_3bits_ip_posid_v18.patch (6.6K, ../../CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw@mail.gmail.com/3-0004_freeup_3bits_ip_posid_v18.patch)
download | inline diff:
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index aa0b02f..1e1c978 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -928,7 +928,7 @@ keyGetItem(GinState *ginstate, MemoryContext tempCtx, GinScanKey key,
* Find the minimum item > advancePast among the active entry streams.
*
* Note: a lossy-page entry is encoded by a ItemPointer with max value for
- * offset (0xffff), so that it will sort after any exact entries for the
+ * offset (0x1fff), so that it will sort after any exact entries for the
* same page. So we'll prefer to return exact pointers not lossy
* pointers, which is good.
*/
diff --git a/src/backend/access/gin/ginpostinglist.c b/src/backend/access/gin/ginpostinglist.c
index 8d2d31a..b22b9f5 100644
--- a/src/backend/access/gin/ginpostinglist.c
+++ b/src/backend/access/gin/ginpostinglist.c
@@ -253,7 +253,7 @@ ginCompressPostingList(const ItemPointer ipd, int nipd, int maxsize,
Assert(ndecoded == totalpacked);
for (i = 0; i < ndecoded; i++)
- Assert(memcmp(&tmp[i], &ipd[i], sizeof(ItemPointerData)) == 0);
+ Assert(ItemPointerEquals(&tmp[i], &ipd[i]));
pfree(tmp);
}
#endif
diff --git a/src/include/access/ginblock.h b/src/include/access/ginblock.h
index 438912c..3f7a3f0 100644
--- a/src/include/access/ginblock.h
+++ b/src/include/access/ginblock.h
@@ -160,14 +160,14 @@ typedef struct GinMetaPageData
(GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0 && \
GinItemPointerGetBlockNumber(p) == (BlockNumber)0)
#define ItemPointerSetMax(p) \
- ItemPointerSet((p), InvalidBlockNumber, (OffsetNumber)0xffff)
+ ItemPointerSet((p), InvalidBlockNumber, (OffsetNumber)OffsetNumberMask)
#define ItemPointerIsMax(p) \
- (GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0xffff && \
+ (GinItemPointerGetOffsetNumber(p) == (OffsetNumber)OffsetNumberMask && \
GinItemPointerGetBlockNumber(p) == InvalidBlockNumber)
#define ItemPointerSetLossyPage(p, b) \
- ItemPointerSet((p), (b), (OffsetNumber)0xffff)
+ ItemPointerSet((p), (b), (OffsetNumber)OffsetNumberMask)
#define ItemPointerIsLossyPage(p) \
- (GinItemPointerGetOffsetNumber(p) == (OffsetNumber)0xffff && \
+ (GinItemPointerGetOffsetNumber(p) == (OffsetNumber)OffsetNumberMask && \
GinItemPointerGetBlockNumber(p) != InvalidBlockNumber)
/*
@@ -218,7 +218,7 @@ typedef signed char GinNullCategory;
*/
#define GinGetNPosting(itup) GinItemPointerGetOffsetNumber(&(itup)->t_tid)
#define GinSetNPosting(itup,n) ItemPointerSetOffsetNumber(&(itup)->t_tid,n)
-#define GIN_TREE_POSTING ((OffsetNumber)0xffff)
+#define GIN_TREE_POSTING ((OffsetNumber)OffsetNumberMask)
#define GinIsPostingTree(itup) (GinGetNPosting(itup) == GIN_TREE_POSTING)
#define GinSetPostingTree(itup, blkno) ( GinSetNPosting((itup),GIN_TREE_POSTING), ItemPointerSetBlockNumber(&(itup)->t_tid, blkno) )
#define GinGetPostingTree(itup) GinItemPointerGetBlockNumber(&(itup)->t_tid)
diff --git a/src/include/access/gist_private.h b/src/include/access/gist_private.h
index 1ad4ed6..0ad11f1 100644
--- a/src/include/access/gist_private.h
+++ b/src/include/access/gist_private.h
@@ -269,8 +269,8 @@ typedef struct
* invalid tuples in an index, so throwing an error is as far as we go with
* supporting that.
*/
-#define TUPLE_IS_VALID 0xffff
-#define TUPLE_IS_INVALID 0xfffe
+#define TUPLE_IS_VALID OffsetNumberMask
+#define TUPLE_IS_INVALID OffsetNumberPrev(OffsetNumberMask)
#define GistTupleIsInvalid(itup) ( ItemPointerGetOffsetNumber( &((itup)->t_tid) ) == TUPLE_IS_INVALID )
#define GistTupleSetValid(itup) ItemPointerSetOffsetNumber( &((itup)->t_tid), TUPLE_IS_VALID )
diff --git a/src/include/access/htup_details.h b/src/include/access/htup_details.h
index 24433c7..4d614b7 100644
--- a/src/include/access/htup_details.h
+++ b/src/include/access/htup_details.h
@@ -288,7 +288,7 @@ struct HeapTupleHeaderData
* than MaxOffsetNumber, so that it can be distinguished from a valid
* offset number in a regular item pointer.
*/
-#define SpecTokenOffsetNumber 0xfffe
+#define SpecTokenOffsetNumber OffsetNumberPrev(OffsetNumberMask)
/*
* HeapTupleHeader accessor macros
diff --git a/src/include/storage/itemptr.h b/src/include/storage/itemptr.h
index 60d0070..3144bdd 100644
--- a/src/include/storage/itemptr.h
+++ b/src/include/storage/itemptr.h
@@ -57,7 +57,7 @@ typedef ItemPointerData *ItemPointer;
* True iff the disk item pointer is not NULL.
*/
#define ItemPointerIsValid(pointer) \
- ((bool) (PointerIsValid(pointer) && ((pointer)->ip_posid != 0)))
+ ((bool) (PointerIsValid(pointer) && (((pointer)->ip_posid & OffsetNumberMask) != 0)))
/*
* ItemPointerGetBlockNumber
@@ -82,13 +82,37 @@ typedef ItemPointerData *ItemPointer;
#define ItemPointerGetOffsetNumber(pointer) \
( \
AssertMacro(ItemPointerIsValid(pointer)), \
- (pointer)->ip_posid \
+ ((pointer)->ip_posid & OffsetNumberMask) \
)
/* Same as ItemPointerGetOffsetNumber but without any assert-checks */
#define ItemPointerGetOffsetNumberNoCheck(pointer) \
( \
- (pointer)->ip_posid \
+ ((pointer)->ip_posid & OffsetNumberMask) \
+)
+
+/*
+ * Get the flags stored in high order bits in the OffsetNumber.
+ */
+#define ItemPointerGetFlags(pointer) \
+( \
+ ((pointer)->ip_posid & ~OffsetNumberMask) >> OffsetNumberBits \
+)
+
+/*
+ * Set the flag bits. We first left-shift since flags are defined starting 0x01
+ */
+#define ItemPointerSetFlags(pointer, flags) \
+( \
+ ((pointer)->ip_posid |= ((flags) << OffsetNumberBits)) \
+)
+
+/*
+ * Clear all flags.
+ */
+#define ItemPointerClearFlags(pointer) \
+( \
+ ((pointer)->ip_posid &= OffsetNumberMask) \
)
/*
@@ -99,7 +123,7 @@ typedef ItemPointerData *ItemPointer;
( \
AssertMacro(PointerIsValid(pointer)), \
BlockIdSet(&((pointer)->ip_blkid), blockNumber), \
- (pointer)->ip_posid = offNum \
+ (pointer)->ip_posid = (offNum) \
)
/*
diff --git a/src/include/storage/off.h b/src/include/storage/off.h
index fe8638f..fe1834c 100644
--- a/src/include/storage/off.h
+++ b/src/include/storage/off.h
@@ -26,8 +26,15 @@ typedef uint16 OffsetNumber;
#define InvalidOffsetNumber ((OffsetNumber) 0)
#define FirstOffsetNumber ((OffsetNumber) 1)
#define MaxOffsetNumber ((OffsetNumber) (BLCKSZ / sizeof(ItemIdData)))
-#define OffsetNumberMask (0xffff) /* valid uint16 bits */
+/*
+ * Currently we support maxinum 32kB blocks and each ItemId takes 6 bytes. That
+ * limits the number of line pointers to (32kB/6 = 5461). 13 bits are enought o
+ * represent all line pointers. Hence we can reuse the high order bits in
+ * OffsetNumber for other purposes.
+ */
+#define OffsetNumberMask (0x1fff) /* valid uint16 bits */
+#define OffsetNumberBits 13 /* number of valid bits in OffsetNumber */
/* ----------------
* support macros
* ----------------
[application/octet-stream] 0003_clear_ip_posid_blkid_refs_v18.patch (11.0K, ../../CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw@mail.gmail.com/4-0003_clear_ip_posid_blkid_refs_v18.patch)
download | inline diff:
diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c
index 6f35e28..07496db 100644
--- a/contrib/pageinspect/btreefuncs.c
+++ b/contrib/pageinspect/btreefuncs.c
@@ -363,8 +363,8 @@ bt_page_items(PG_FUNCTION_ARGS)
j = 0;
values[j++] = psprintf("%d", uargs->offset);
values[j++] = psprintf("(%u,%u)",
- BlockIdGetBlockNumber(&(itup->t_tid.ip_blkid)),
- itup->t_tid.ip_posid);
+ ItemPointerGetBlockNumberNoCheck(&itup->t_tid),
+ ItemPointerGetOffsetNumberNoCheck(&itup->t_tid));
values[j++] = psprintf("%d", (int) IndexTupleSize(itup));
values[j++] = psprintf("%c", IndexTupleHasNulls(itup) ? 't' : 'f');
values[j++] = psprintf("%c", IndexTupleHasVarwidths(itup) ? 't' : 'f');
diff --git a/contrib/pgstattuple/pgstattuple.c b/contrib/pgstattuple/pgstattuple.c
index 1e0de5d..44f90cd 100644
--- a/contrib/pgstattuple/pgstattuple.c
+++ b/contrib/pgstattuple/pgstattuple.c
@@ -356,7 +356,7 @@ pgstat_heap(Relation rel, FunctionCallInfo fcinfo)
* heap_getnext may find no tuples on a given page, so we cannot
* simply examine the pages returned by the heap scan.
*/
- tupblock = BlockIdGetBlockNumber(&tuple->t_self.ip_blkid);
+ tupblock = ItemPointerGetBlockNumber(&tuple->t_self);
while (block <= tupblock)
{
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 87cd9ea..aa0b02f 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -626,8 +626,9 @@ entryLoadMoreItems(GinState *ginstate, GinScanEntry entry,
}
else
{
- entry->btree.itemptr = advancePast;
- entry->btree.itemptr.ip_posid++;
+ ItemPointerSet(&entry->btree.itemptr,
+ GinItemPointerGetBlockNumber(&advancePast),
+ OffsetNumberNext(GinItemPointerGetOffsetNumber(&advancePast)));
}
entry->btree.fullScan = false;
stack = ginFindLeafPage(&entry->btree, true, snapshot);
@@ -979,15 +980,17 @@ keyGetItem(GinState *ginstate, MemoryContext tempCtx, GinScanKey key,
if (GinItemPointerGetBlockNumber(&advancePast) <
GinItemPointerGetBlockNumber(&minItem))
{
- advancePast.ip_blkid = minItem.ip_blkid;
- advancePast.ip_posid = 0;
+ ItemPointerSet(&advancePast,
+ GinItemPointerGetBlockNumber(&minItem),
+ InvalidOffsetNumber);
}
}
else
{
- Assert(minItem.ip_posid > 0);
- advancePast = minItem;
- advancePast.ip_posid--;
+ Assert(GinItemPointerGetOffsetNumber(&minItem) > 0);
+ ItemPointerSet(&advancePast,
+ GinItemPointerGetBlockNumber(&minItem),
+ OffsetNumberPrev(GinItemPointerGetOffsetNumber(&minItem)));
}
/*
@@ -1245,15 +1248,17 @@ scanGetItem(IndexScanDesc scan, ItemPointerData advancePast,
if (GinItemPointerGetBlockNumber(&advancePast) <
GinItemPointerGetBlockNumber(&key->curItem))
{
- advancePast.ip_blkid = key->curItem.ip_blkid;
- advancePast.ip_posid = 0;
+ ItemPointerSet(&advancePast,
+ GinItemPointerGetBlockNumber(&key->curItem),
+ InvalidOffsetNumber);
}
}
else
{
- Assert(key->curItem.ip_posid > 0);
- advancePast = key->curItem;
- advancePast.ip_posid--;
+ Assert(GinItemPointerGetOffsetNumber(&key->curItem) > 0);
+ ItemPointerSet(&advancePast,
+ GinItemPointerGetBlockNumber(&key->curItem),
+ OffsetNumberPrev(GinItemPointerGetOffsetNumber(&key->curItem)));
}
/*
diff --git a/src/backend/access/gin/ginpostinglist.c b/src/backend/access/gin/ginpostinglist.c
index 598069d..8d2d31a 100644
--- a/src/backend/access/gin/ginpostinglist.c
+++ b/src/backend/access/gin/ginpostinglist.c
@@ -79,13 +79,11 @@ itemptr_to_uint64(const ItemPointer iptr)
uint64 val;
Assert(ItemPointerIsValid(iptr));
- Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits));
+ Assert(GinItemPointerGetOffsetNumber(iptr) < (1 << MaxHeapTuplesPerPageBits));
- val = iptr->ip_blkid.bi_hi;
- val <<= 16;
- val |= iptr->ip_blkid.bi_lo;
+ val = GinItemPointerGetBlockNumber(iptr);
val <<= MaxHeapTuplesPerPageBits;
- val |= iptr->ip_posid;
+ val |= GinItemPointerGetOffsetNumber(iptr);
return val;
}
@@ -93,11 +91,9 @@ itemptr_to_uint64(const ItemPointer iptr)
static inline void
uint64_to_itemptr(uint64 val, ItemPointer iptr)
{
- iptr->ip_posid = val & ((1 << MaxHeapTuplesPerPageBits) - 1);
+ GinItemPointerSetOffsetNumber(iptr, val & ((1 << MaxHeapTuplesPerPageBits) - 1));
val = val >> MaxHeapTuplesPerPageBits;
- iptr->ip_blkid.bi_lo = val & 0xFFFF;
- val = val >> 16;
- iptr->ip_blkid.bi_hi = val & 0xFFFF;
+ GinItemPointerSetBlockNumber(iptr, val);
Assert(ItemPointerIsValid(iptr));
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b437799..12ebadc 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3013,8 +3013,8 @@ DisplayMapping(HTAB *tuplecid_data)
ent->key.relnode.dbNode,
ent->key.relnode.spcNode,
ent->key.relnode.relNode,
- BlockIdGetBlockNumber(&ent->key.tid.ip_blkid),
- ent->key.tid.ip_posid,
+ ItemPointerGetBlockNumber(&ent->key.tid),
+ ItemPointerGetOffsetNumber(&ent->key.tid),
ent->cmin,
ent->cmax
);
diff --git a/src/backend/storage/page/itemptr.c b/src/backend/storage/page/itemptr.c
index 703cbb9..28ac885 100644
--- a/src/backend/storage/page/itemptr.c
+++ b/src/backend/storage/page/itemptr.c
@@ -54,18 +54,21 @@ ItemPointerCompare(ItemPointer arg1, ItemPointer arg2)
/*
* Don't use ItemPointerGetBlockNumber or ItemPointerGetOffsetNumber here,
* because they assert ip_posid != 0 which might not be true for a
- * user-supplied TID.
+ * user-supplied TID. Instead we use ItemPointerGetBlockNumberNoCheck and
+ * ItemPointerGetOffsetNumberNoCheck which do not do any validation.
*/
- BlockNumber b1 = BlockIdGetBlockNumber(&(arg1->ip_blkid));
- BlockNumber b2 = BlockIdGetBlockNumber(&(arg2->ip_blkid));
+ BlockNumber b1 = ItemPointerGetBlockNumberNoCheck(arg1);
+ BlockNumber b2 = ItemPointerGetBlockNumberNoCheck(arg2);
if (b1 < b2)
return -1;
else if (b1 > b2)
return 1;
- else if (arg1->ip_posid < arg2->ip_posid)
+ else if (ItemPointerGetOffsetNumberNoCheck(arg1) <
+ ItemPointerGetOffsetNumberNoCheck(arg2))
return -1;
- else if (arg1->ip_posid > arg2->ip_posid)
+ else if (ItemPointerGetOffsetNumberNoCheck(arg1) >
+ ItemPointerGetOffsetNumberNoCheck(arg2))
return 1;
else
return 0;
diff --git a/src/backend/utils/adt/tid.c b/src/backend/utils/adt/tid.c
index 49a5a15..7f3a692 100644
--- a/src/backend/utils/adt/tid.c
+++ b/src/backend/utils/adt/tid.c
@@ -109,8 +109,8 @@ tidout(PG_FUNCTION_ARGS)
OffsetNumber offsetNumber;
char buf[32];
- blockNumber = BlockIdGetBlockNumber(&(itemPtr->ip_blkid));
- offsetNumber = itemPtr->ip_posid;
+ blockNumber = ItemPointerGetBlockNumberNoCheck(itemPtr);
+ offsetNumber = ItemPointerGetOffsetNumberNoCheck(itemPtr);
/* Perhaps someday we should output this as a record. */
snprintf(buf, sizeof(buf), "(%u,%u)", blockNumber, offsetNumber);
@@ -146,14 +146,12 @@ Datum
tidsend(PG_FUNCTION_ARGS)
{
ItemPointer itemPtr = PG_GETARG_ITEMPOINTER(0);
- BlockId blockId;
BlockNumber blockNumber;
OffsetNumber offsetNumber;
StringInfoData buf;
- blockId = &(itemPtr->ip_blkid);
- blockNumber = BlockIdGetBlockNumber(blockId);
- offsetNumber = itemPtr->ip_posid;
+ blockNumber = ItemPointerGetBlockNumberNoCheck(itemPtr);
+ offsetNumber = ItemPointerGetOffsetNumberNoCheck(itemPtr);
pq_begintypsend(&buf);
pq_sendint(&buf, blockNumber, sizeof(blockNumber));
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 34e7339..2fd4479 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -460,8 +460,8 @@ extern ItemPointer ginMergeItemPointers(ItemPointerData *a, uint32 na,
static inline int
ginCompareItemPointers(ItemPointer a, ItemPointer b)
{
- uint64 ia = (uint64) a->ip_blkid.bi_hi << 32 | (uint64) a->ip_blkid.bi_lo << 16 | a->ip_posid;
- uint64 ib = (uint64) b->ip_blkid.bi_hi << 32 | (uint64) b->ip_blkid.bi_lo << 16 | b->ip_posid;
+ uint64 ia = (uint64) GinItemPointerGetBlockNumber(a) << 32 | GinItemPointerGetOffsetNumber(a);
+ uint64 ib = (uint64) GinItemPointerGetBlockNumber(b) << 32 | GinItemPointerGetOffsetNumber(b);
if (ia == ib)
return 0;
diff --git a/src/include/access/ginblock.h b/src/include/access/ginblock.h
index a3fb056..438912c 100644
--- a/src/include/access/ginblock.h
+++ b/src/include/access/ginblock.h
@@ -132,10 +132,17 @@ typedef struct GinMetaPageData
* to avoid Asserts, since sometimes the ip_posid isn't "valid"
*/
#define GinItemPointerGetBlockNumber(pointer) \
- BlockIdGetBlockNumber(&(pointer)->ip_blkid)
+ (ItemPointerGetBlockNumberNoCheck(pointer))
#define GinItemPointerGetOffsetNumber(pointer) \
- ((pointer)->ip_posid)
+ (ItemPointerGetOffsetNumberNoCheck(pointer))
+
+#define GinItemPointerSetBlockNumber(pointer, blkno) \
+ (ItemPointerSetBlockNumber((pointer), (blkno)))
+
+#define GinItemPointerSetOffsetNumber(pointer, offnum) \
+ (ItemPointerSetOffsetNumber((pointer), (offnum)))
+
/*
* Special-case item pointer values needed by the GIN search logic.
diff --git a/src/include/access/htup_details.h b/src/include/access/htup_details.h
index 7552186..24433c7 100644
--- a/src/include/access/htup_details.h
+++ b/src/include/access/htup_details.h
@@ -428,7 +428,7 @@ do { \
#define HeapTupleHeaderIsSpeculative(tup) \
( \
- (tup)->t_ctid.ip_posid == SpecTokenOffsetNumber \
+ (ItemPointerGetOffsetNumberNoCheck(&(tup)->t_ctid) == SpecTokenOffsetNumber) \
)
#define HeapTupleHeaderGetSpeculativeToken(tup) \
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index 6289ffa..f9304db 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -151,9 +151,8 @@ typedef struct BTMetaPageData
* within a level). - vadim 04/09/97
*/
#define BTTidSame(i1, i2) \
- ( (i1).ip_blkid.bi_hi == (i2).ip_blkid.bi_hi && \
- (i1).ip_blkid.bi_lo == (i2).ip_blkid.bi_lo && \
- (i1).ip_posid == (i2).ip_posid )
+ ((ItemPointerGetBlockNumber(&(i1)) == ItemPointerGetBlockNumber(&(i2))) && \
+ (ItemPointerGetOffsetNumber(&(i1)) == ItemPointerGetOffsetNumber(&(i2))))
#define BTEntrySame(i1, i2) \
BTTidSame((i1)->t_tid, (i2)->t_tid)
diff --git a/src/include/storage/itemptr.h b/src/include/storage/itemptr.h
index 576aaa8..60d0070 100644
--- a/src/include/storage/itemptr.h
+++ b/src/include/storage/itemptr.h
@@ -69,6 +69,12 @@ typedef ItemPointerData *ItemPointer;
BlockIdGetBlockNumber(&(pointer)->ip_blkid) \
)
+/* Same as ItemPointerGetBlockNumber but without any assert-checks */
+#define ItemPointerGetBlockNumberNoCheck(pointer) \
+( \
+ BlockIdGetBlockNumber(&(pointer)->ip_blkid) \
+)
+
/*
* ItemPointerGetOffsetNumber
* Returns the offset number of a disk item pointer.
@@ -79,6 +85,12 @@ typedef ItemPointerData *ItemPointer;
(pointer)->ip_posid \
)
+/* Same as ItemPointerGetOffsetNumber but without any assert-checks */
+#define ItemPointerGetOffsetNumberNoCheck(pointer) \
+( \
+ (pointer)->ip_posid \
+)
+
/*
* ItemPointerSet
* Sets a disk item pointer to the specified block and offset.
[application/octet-stream] 0002_track_root_lp_v18.patch (38.4K, ../../CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw@mail.gmail.com/5-0002_track_root_lp_v18.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index fd4291b..26a7af4 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -94,7 +94,8 @@ static HeapTuple heap_prepare_insert(Relation relation, HeapTuple tup,
TransactionId xid, CommandId cid, int options);
static XLogRecPtr log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup,
- HeapTuple newtup, HeapTuple old_key_tup,
+ HeapTuple newtup, OffsetNumber root_offnum,
+ HeapTuple old_key_tup,
bool all_visible_cleared, bool new_all_visible_cleared);
static Bitmapset *HeapDetermineModifiedColumns(Relation relation,
Bitmapset *interesting_cols,
@@ -2264,13 +2265,13 @@ heap_get_latest_tid(Relation relation,
*/
if ((tp.t_data->t_infomask & HEAP_XMAX_INVALID) ||
HeapTupleHeaderIsOnlyLocked(tp.t_data) ||
- ItemPointerEquals(&tp.t_self, &tp.t_data->t_ctid))
+ HeapTupleHeaderIsHeapLatest(tp.t_data, &ctid))
{
UnlockReleaseBuffer(buffer);
break;
}
- ctid = tp.t_data->t_ctid;
+ HeapTupleHeaderGetNextTid(tp.t_data, &ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(tp.t_data);
UnlockReleaseBuffer(buffer);
} /* end of loop */
@@ -2401,6 +2402,7 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
Buffer buffer;
Buffer vmbuffer = InvalidBuffer;
bool all_visible_cleared = false;
+ OffsetNumber root_offnum;
/*
* Fill in tuple header fields, assign an OID, and toast the tuple if
@@ -2439,8 +2441,13 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
/* NO EREPORT(ERROR) from here till changes are logged */
START_CRIT_SECTION();
- RelationPutHeapTuple(relation, buffer, heaptup,
- (options & HEAP_INSERT_SPECULATIVE) != 0);
+ root_offnum = RelationPutHeapTuple(relation, buffer, heaptup,
+ (options & HEAP_INSERT_SPECULATIVE) != 0,
+ InvalidOffsetNumber);
+
+ /* We must not overwrite the speculative insertion token. */
+ if ((options & HEAP_INSERT_SPECULATIVE) == 0)
+ HeapTupleHeaderSetHeapLatest(heaptup->t_data, root_offnum);
if (PageIsAllVisible(BufferGetPage(buffer)))
{
@@ -2668,6 +2675,7 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
Size saveFreeSpace;
bool need_tuple_data = RelationIsLogicallyLogged(relation);
bool need_cids = RelationIsAccessibleInLogicalDecoding(relation);
+ OffsetNumber root_offnum;
needwal = !(options & HEAP_INSERT_SKIP_WAL) && RelationNeedsWAL(relation);
saveFreeSpace = RelationGetTargetPageFreeSpace(relation,
@@ -2738,7 +2746,12 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
* RelationGetBufferForTuple has ensured that the first tuple fits.
* Put that on the page, and then as many other tuples as fit.
*/
- RelationPutHeapTuple(relation, buffer, heaptuples[ndone], false);
+ root_offnum = RelationPutHeapTuple(relation, buffer, heaptuples[ndone], false,
+ InvalidOffsetNumber);
+
+ /* Mark this tuple as the latest and also set root offset. */
+ HeapTupleHeaderSetHeapLatest(heaptuples[ndone]->t_data, root_offnum);
+
for (nthispage = 1; ndone + nthispage < ntuples; nthispage++)
{
HeapTuple heaptup = heaptuples[ndone + nthispage];
@@ -2746,7 +2759,10 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
if (PageGetHeapFreeSpace(page) < MAXALIGN(heaptup->t_len) + saveFreeSpace)
break;
- RelationPutHeapTuple(relation, buffer, heaptup, false);
+ root_offnum = RelationPutHeapTuple(relation, buffer, heaptup, false,
+ InvalidOffsetNumber);
+ /* Mark each tuple as the latest and also set root offset. */
+ HeapTupleHeaderSetHeapLatest(heaptup->t_data, root_offnum);
/*
* We don't use heap_multi_insert for catalog tuples yet, but
@@ -3018,6 +3034,7 @@ heap_delete(Relation relation, ItemPointer tid,
HeapTupleData tp;
Page page;
BlockNumber block;
+ OffsetNumber offnum;
Buffer buffer;
Buffer vmbuffer = InvalidBuffer;
TransactionId new_xmax;
@@ -3028,6 +3045,7 @@ heap_delete(Relation relation, ItemPointer tid,
bool all_visible_cleared = false;
HeapTuple old_key_tuple = NULL; /* replica identity of the tuple */
bool old_key_copied = false;
+ OffsetNumber root_offnum;
Assert(ItemPointerIsValid(tid));
@@ -3069,7 +3087,8 @@ heap_delete(Relation relation, ItemPointer tid,
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
}
- lp = PageGetItemId(page, ItemPointerGetOffsetNumber(tid));
+ offnum = ItemPointerGetOffsetNumber(tid);
+ lp = PageGetItemId(page, offnum);
Assert(ItemIdIsNormal(lp));
tp.t_tableOid = RelationGetRelid(relation);
@@ -3199,7 +3218,17 @@ l1:
result == HeapTupleUpdated ||
result == HeapTupleBeingUpdated);
Assert(!(tp.t_data->t_infomask & HEAP_XMAX_INVALID));
- hufd->ctid = tp.t_data->t_ctid;
+
+ /*
+ * If we're at the end of the chain, then just return the same TID back
+ * to the caller. The caller uses that as a hint to know if we have hit
+ * the end of the chain.
+ */
+ if (!HeapTupleHeaderIsHeapLatest(tp.t_data, &tp.t_self))
+ HeapTupleHeaderGetNextTid(tp.t_data, &hufd->ctid);
+ else
+ ItemPointerCopy(&tp.t_self, &hufd->ctid);
+
hufd->xmax = HeapTupleHeaderGetUpdateXid(tp.t_data);
if (result == HeapTupleSelfUpdated)
hufd->cmax = HeapTupleHeaderGetCmax(tp.t_data);
@@ -3248,6 +3277,22 @@ l1:
xid, LockTupleExclusive, true,
&new_xmax, &new_infomask, &new_infomask2);
+ /*
+ * heap_get_root_tuple_one() may call palloc, which is disallowed once we
+ * enter the critical section. So check if the root offset is cached in the
+ * tuple and if not, fetch that information hard way before entering the
+ * critical section.
+ *
+ * Most often and unless we are dealing with a pg-upgraded cluster, the
+ * root offset information should be cached. So there should not be too
+ * much overhead of fetching this information. Also, once a tuple is
+ * updated, the information will be copied to the new version. So it's not
+ * as if we're going to pay this price forever.
+ */
+ if (!HeapTupleHeaderHasRootOffset(tp.t_data))
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&tp.t_self));
+
START_CRIT_SECTION();
/*
@@ -3275,8 +3320,10 @@ l1:
HeapTupleHeaderClearHotUpdated(tp.t_data);
HeapTupleHeaderSetXmax(tp.t_data, new_xmax);
HeapTupleHeaderSetCmax(tp.t_data, cid, iscombo);
- /* Make sure there is no forward chain link in t_ctid */
- tp.t_data->t_ctid = tp.t_self;
+
+ /* Mark this tuple as the latest tuple in the update chain. */
+ if (!HeapTupleHeaderHasRootOffset(tp.t_data))
+ HeapTupleHeaderSetHeapLatest(tp.t_data, root_offnum);
MarkBufferDirty(buffer);
@@ -3477,6 +3524,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
bool old_key_copied = false;
Page page;
BlockNumber block;
+ OffsetNumber offnum;
+ OffsetNumber root_offnum;
MultiXactStatus mxact_status;
Buffer buffer,
newbuf,
@@ -3539,6 +3588,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
block = ItemPointerGetBlockNumber(otid);
+ offnum = ItemPointerGetOffsetNumber(otid);
buffer = ReadBuffer(relation, block);
page = BufferGetPage(buffer);
@@ -3823,7 +3873,12 @@ l2:
result == HeapTupleUpdated ||
result == HeapTupleBeingUpdated);
Assert(!(oldtup.t_data->t_infomask & HEAP_XMAX_INVALID));
- hufd->ctid = oldtup.t_data->t_ctid;
+
+ if (!HeapTupleHeaderIsHeapLatest(oldtup.t_data, &oldtup.t_self))
+ HeapTupleHeaderGetNextTid(oldtup.t_data, &hufd->ctid);
+ else
+ ItemPointerCopy(&oldtup.t_self, &hufd->ctid);
+
hufd->xmax = HeapTupleHeaderGetUpdateXid(oldtup.t_data);
if (result == HeapTupleSelfUpdated)
hufd->cmax = HeapTupleHeaderGetCmax(oldtup.t_data);
@@ -3963,6 +4018,7 @@ l2:
uint16 infomask_lock_old_tuple,
infomask2_lock_old_tuple;
bool cleared_all_frozen = false;
+ OffsetNumber root_offnum;
/*
* To prevent concurrent sessions from updating the tuple, we have to
@@ -3990,6 +4046,14 @@ l2:
Assert(HEAP_XMAX_IS_LOCKED_ONLY(infomask_lock_old_tuple));
+ /*
+ * Fetch root offset before entering the critical section. We do this
+ * only if the information is not already available.
+ */
+ if (!HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&oldtup.t_self));
+
START_CRIT_SECTION();
/* Clear obsolete visibility flags ... */
@@ -4004,7 +4068,8 @@ l2:
HeapTupleHeaderSetCmax(oldtup.t_data, cid, iscombo);
/* temporarily make it look not-updated, but locked */
- oldtup.t_data->t_ctid = oldtup.t_self;
+ if (!HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ HeapTupleHeaderSetHeapLatest(oldtup.t_data, root_offnum);
/*
* Clear all-frozen bit on visibility map if needed. We could
@@ -4162,6 +4227,10 @@ l2:
bms_overlap(modified_attrs, id_attrs),
&old_key_copied);
+ if (!HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&(oldtup.t_self)));
+
/* NO EREPORT(ERROR) from here till changes are logged */
START_CRIT_SECTION();
@@ -4187,6 +4256,17 @@ l2:
HeapTupleSetHeapOnly(heaptup);
/* Mark the caller's copy too, in case different from heaptup */
HeapTupleSetHeapOnly(newtup);
+ /*
+ * For HOT (or WARM) updated tuples, we store the offset of the root
+ * line pointer of this chain in the ip_posid field of the new tuple.
+ * Usually this information will be available in the corresponding
+ * field of the old tuple. But for aborted updates or pg_upgraded
+ * databases, we might be seeing the old-style CTID chains and hence
+ * the information must be obtained by hard way (we should have done
+ * that before entering the critical section above).
+ */
+ if (HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ root_offnum = HeapTupleHeaderGetRootOffset(oldtup.t_data);
}
else
{
@@ -4194,10 +4274,22 @@ l2:
HeapTupleClearHotUpdated(&oldtup);
HeapTupleClearHeapOnly(heaptup);
HeapTupleClearHeapOnly(newtup);
+ root_offnum = InvalidOffsetNumber;
}
- RelationPutHeapTuple(relation, newbuf, heaptup, false); /* insert new tuple */
-
+ /* insert new tuple */
+ root_offnum = RelationPutHeapTuple(relation, newbuf, heaptup, false,
+ root_offnum);
+ /*
+ * Also mark both copies as latest and set the root offset information. If
+ * we're doing a HOT/WARM update, then we just copy the information from
+ * old tuple, if available or computed above. For regular updates,
+ * RelationPutHeapTuple must have returned us the actual offset number
+ * where the new version was inserted and we store the same value since the
+ * update resulted in a new HOT-chain.
+ */
+ HeapTupleHeaderSetHeapLatest(heaptup->t_data, root_offnum);
+ HeapTupleHeaderSetHeapLatest(newtup->t_data, root_offnum);
/* Clear obsolete visibility flags, possibly set by ourselves above... */
oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
@@ -4210,7 +4302,7 @@ l2:
HeapTupleHeaderSetCmax(oldtup.t_data, cid, iscombo);
/* record address of new tuple in t_ctid of old one */
- oldtup.t_data->t_ctid = heaptup->t_self;
+ HeapTupleHeaderSetNextTid(oldtup.t_data, &(heaptup->t_self));
/* clear PD_ALL_VISIBLE flags, reset all visibilitymap bits */
if (PageIsAllVisible(BufferGetPage(buffer)))
@@ -4249,6 +4341,7 @@ l2:
recptr = log_heap_update(relation, buffer,
newbuf, &oldtup, heaptup,
+ root_offnum,
old_key_tuple,
all_visible_cleared,
all_visible_cleared_new);
@@ -4529,7 +4622,8 @@ heap_lock_tuple(Relation relation, HeapTuple tuple,
ItemId lp;
Page page;
Buffer vmbuffer = InvalidBuffer;
- BlockNumber block;
+ BlockNumber block;
+ OffsetNumber offnum;
TransactionId xid,
xmax;
uint16 old_infomask,
@@ -4538,9 +4632,11 @@ heap_lock_tuple(Relation relation, HeapTuple tuple,
bool first_time = true;
bool have_tuple_lock = false;
bool cleared_all_frozen = false;
+ OffsetNumber root_offnum;
*buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
block = ItemPointerGetBlockNumber(tid);
+ offnum = ItemPointerGetOffsetNumber(tid);
/*
* Before locking the buffer, pin the visibility map page if it appears to
@@ -4560,6 +4656,7 @@ heap_lock_tuple(Relation relation, HeapTuple tuple,
tuple->t_data = (HeapTupleHeader) PageGetItem(page, lp);
tuple->t_len = ItemIdGetLength(lp);
tuple->t_tableOid = RelationGetRelid(relation);
+ tuple->t_self = *tid;
l3:
result = HeapTupleSatisfiesUpdate(tuple, cid, *buffer);
@@ -4587,7 +4684,11 @@ l3:
xwait = HeapTupleHeaderGetRawXmax(tuple->t_data);
infomask = tuple->t_data->t_infomask;
infomask2 = tuple->t_data->t_infomask2;
- ItemPointerCopy(&tuple->t_data->t_ctid, &t_ctid);
+
+ if (!HeapTupleHeaderIsHeapLatest(tuple->t_data, tid))
+ HeapTupleHeaderGetNextTid(tuple->t_data, &t_ctid);
+ else
+ ItemPointerCopy(tid, &t_ctid);
LockBuffer(*buffer, BUFFER_LOCK_UNLOCK);
@@ -5025,7 +5126,12 @@ failed:
Assert(result == HeapTupleSelfUpdated || result == HeapTupleUpdated ||
result == HeapTupleWouldBlock);
Assert(!(tuple->t_data->t_infomask & HEAP_XMAX_INVALID));
- hufd->ctid = tuple->t_data->t_ctid;
+
+ if (!HeapTupleHeaderIsHeapLatest(tuple->t_data, tid))
+ HeapTupleHeaderGetNextTid(tuple->t_data, &hufd->ctid);
+ else
+ ItemPointerCopy(tid, &hufd->ctid);
+
hufd->xmax = HeapTupleHeaderGetUpdateXid(tuple->t_data);
if (result == HeapTupleSelfUpdated)
hufd->cmax = HeapTupleHeaderGetCmax(tuple->t_data);
@@ -5073,6 +5179,10 @@ failed:
GetCurrentTransactionId(), mode, false,
&xid, &new_infomask, &new_infomask2);
+ if (!HeapTupleHeaderHasRootOffset(tuple->t_data))
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&tuple->t_self));
+
START_CRIT_SECTION();
/*
@@ -5101,7 +5211,10 @@ failed:
* the tuple as well.
*/
if (HEAP_XMAX_IS_LOCKED_ONLY(new_infomask))
- tuple->t_data->t_ctid = *tid;
+ {
+ if (!HeapTupleHeaderHasRootOffset(tuple->t_data))
+ HeapTupleHeaderSetHeapLatest(tuple->t_data, root_offnum);
+ }
/* Clear only the all-frozen bit on visibility map if needed */
if (PageIsAllVisible(page) &&
@@ -5615,6 +5728,7 @@ heap_lock_updated_tuple_rec(Relation rel, ItemPointer tid, TransactionId xid,
bool cleared_all_frozen = false;
Buffer vmbuffer = InvalidBuffer;
BlockNumber block;
+ OffsetNumber offnum;
ItemPointerCopy(tid, &tupid);
@@ -5623,6 +5737,8 @@ heap_lock_updated_tuple_rec(Relation rel, ItemPointer tid, TransactionId xid,
new_infomask = 0;
new_xmax = InvalidTransactionId;
block = ItemPointerGetBlockNumber(&tupid);
+ offnum = ItemPointerGetOffsetNumber(&tupid);
+
ItemPointerCopy(&tupid, &(mytup.t_self));
if (!heap_fetch(rel, SnapshotAny, &mytup, &buf, false, NULL))
@@ -5852,7 +5968,7 @@ l4:
/* if we find the end of update chain, we're done. */
if (mytup.t_data->t_infomask & HEAP_XMAX_INVALID ||
- ItemPointerEquals(&mytup.t_self, &mytup.t_data->t_ctid) ||
+ HeapTupleHeaderIsHeapLatest(mytup.t_data, &mytup.t_self) ||
HeapTupleHeaderIsOnlyLocked(mytup.t_data))
{
result = HeapTupleMayBeUpdated;
@@ -5861,7 +5977,7 @@ l4:
/* tail recursion */
priorXmax = HeapTupleHeaderGetUpdateXid(mytup.t_data);
- ItemPointerCopy(&(mytup.t_data->t_ctid), &tupid);
+ HeapTupleHeaderGetNextTid(mytup.t_data, &tupid);
UnlockReleaseBuffer(buf);
if (vmbuffer != InvalidBuffer)
ReleaseBuffer(vmbuffer);
@@ -5978,7 +6094,7 @@ heap_finish_speculative(Relation relation, HeapTuple tuple)
* Replace the speculative insertion token with a real t_ctid, pointing to
* itself like it does on regular tuples.
*/
- htup->t_ctid = tuple->t_self;
+ HeapTupleHeaderSetHeapLatest(htup, offnum);
/* XLOG stuff */
if (RelationNeedsWAL(relation))
@@ -6104,8 +6220,7 @@ heap_abort_speculative(Relation relation, HeapTuple tuple)
HeapTupleHeaderSetXmin(tp.t_data, InvalidTransactionId);
/* Clear the speculative insertion token too */
- tp.t_data->t_ctid = tp.t_self;
-
+ HeapTupleHeaderSetHeapLatest(tp.t_data, ItemPointerGetOffsetNumber(tid));
MarkBufferDirty(buffer);
/*
@@ -7453,6 +7568,7 @@ log_heap_visible(RelFileNode rnode, Buffer heap_buffer, Buffer vm_buffer,
static XLogRecPtr
log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup, HeapTuple newtup,
+ OffsetNumber root_offnum,
HeapTuple old_key_tuple,
bool all_visible_cleared, bool new_all_visible_cleared)
{
@@ -7573,6 +7689,9 @@ log_heap_update(Relation reln, Buffer oldbuf,
xlrec.new_offnum = ItemPointerGetOffsetNumber(&newtup->t_self);
xlrec.new_xmax = HeapTupleHeaderGetRawXmax(newtup->t_data);
+ Assert(OffsetNumberIsValid(root_offnum));
+ xlrec.root_offnum = root_offnum;
+
bufflags = REGBUF_STANDARD;
if (init)
bufflags |= REGBUF_WILL_INIT;
@@ -8227,7 +8346,13 @@ heap_xlog_delete(XLogReaderState *record)
PageClearAllVisible(page);
/* Make sure there is no forward chain link in t_ctid */
- htup->t_ctid = target_tid;
+ if (!HeapTupleHeaderHasRootOffset(htup))
+ {
+ OffsetNumber root_offnum;
+ root_offnum = heap_get_root_tuple(page, xlrec->offnum);
+ HeapTupleHeaderSetHeapLatest(htup, root_offnum);
+ }
+
PageSetLSN(page, lsn);
MarkBufferDirty(buffer);
}
@@ -8317,7 +8442,8 @@ heap_xlog_insert(XLogReaderState *record)
htup->t_hoff = xlhdr.t_hoff;
HeapTupleHeaderSetXmin(htup, XLogRecGetXid(record));
HeapTupleHeaderSetCmin(htup, FirstCommandId);
- htup->t_ctid = target_tid;
+
+ HeapTupleHeaderSetHeapLatest(htup, xlrec->offnum);
if (PageAddItem(page, (Item) htup, newlen, xlrec->offnum,
true, true) == InvalidOffsetNumber)
@@ -8452,8 +8578,8 @@ heap_xlog_multi_insert(XLogReaderState *record)
htup->t_hoff = xlhdr->t_hoff;
HeapTupleHeaderSetXmin(htup, XLogRecGetXid(record));
HeapTupleHeaderSetCmin(htup, FirstCommandId);
- ItemPointerSetBlockNumber(&htup->t_ctid, blkno);
- ItemPointerSetOffsetNumber(&htup->t_ctid, offnum);
+
+ HeapTupleHeaderSetHeapLatest(htup, offnum);
offnum = PageAddItem(page, (Item) htup, newlen, offnum, true, true);
if (offnum == InvalidOffsetNumber)
@@ -8589,7 +8715,7 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
HeapTupleHeaderSetXmax(htup, xlrec->old_xmax);
HeapTupleHeaderSetCmax(htup, FirstCommandId, false);
/* Set forward chain link in t_ctid */
- htup->t_ctid = newtid;
+ HeapTupleHeaderSetNextTid(htup, &newtid);
/* Mark the page as a candidate for pruning */
PageSetPrunable(page, XLogRecGetXid(record));
@@ -8722,13 +8848,17 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
HeapTupleHeaderSetXmin(htup, XLogRecGetXid(record));
HeapTupleHeaderSetCmin(htup, FirstCommandId);
HeapTupleHeaderSetXmax(htup, xlrec->new_xmax);
- /* Make sure there is no forward chain link in t_ctid */
- htup->t_ctid = newtid;
offnum = PageAddItem(page, (Item) htup, newlen, offnum, true, true);
if (offnum == InvalidOffsetNumber)
elog(PANIC, "failed to add tuple");
+ /*
+ * Make sure the tuple is marked as the latest and root offset
+ * information is restored.
+ */
+ HeapTupleHeaderSetHeapLatest(htup, xlrec->root_offnum);
+
if (xlrec->flags & XLH_UPDATE_NEW_ALL_VISIBLE_CLEARED)
PageClearAllVisible(page);
@@ -8791,6 +8921,9 @@ heap_xlog_confirm(XLogReaderState *record)
*/
ItemPointerSet(&htup->t_ctid, BufferGetBlockNumber(buffer), offnum);
+ /* For newly inserted tuple, set root offset to itself. */
+ HeapTupleHeaderSetHeapLatest(htup, offnum);
+
PageSetLSN(page, lsn);
MarkBufferDirty(buffer);
}
@@ -8854,11 +8987,17 @@ heap_xlog_lock(XLogReaderState *record)
*/
if (HEAP_XMAX_IS_LOCKED_ONLY(htup->t_infomask))
{
+ ItemPointerData target_tid;
+
+ ItemPointerSet(&target_tid, BufferGetBlockNumber(buffer), offnum);
HeapTupleHeaderClearHotUpdated(htup);
/* Make sure there is no forward chain link in t_ctid */
- ItemPointerSet(&htup->t_ctid,
- BufferGetBlockNumber(buffer),
- offnum);
+ if (!HeapTupleHeaderHasRootOffset(htup))
+ {
+ OffsetNumber root_offnum;
+ root_offnum = heap_get_root_tuple(page, offnum);
+ HeapTupleHeaderSetHeapLatest(htup, root_offnum);
+ }
}
HeapTupleHeaderSetXmax(htup, xlrec->locking_xid);
HeapTupleHeaderSetCmax(htup, FirstCommandId, false);
diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c
index 6529fe3..8052519 100644
--- a/src/backend/access/heap/hio.c
+++ b/src/backend/access/heap/hio.c
@@ -31,12 +31,20 @@
* !!! EREPORT(ERROR) IS DISALLOWED HERE !!! Must PANIC on failure!!!
*
* Note - caller must hold BUFFER_LOCK_EXCLUSIVE on the buffer.
+ *
+ * The caller can optionally tell us to set the root offset to the given value.
+ * Otherwise, the root offset is set to the offset of the new location once its
+ * known. The former is used while updating an existing tuple where the caller
+ * tells us about the root line pointer of the chain. The latter is used
+ * during insertion of a new row, hence root line pointer is set to the offset
+ * where this tuple is inserted.
*/
-void
+OffsetNumber
RelationPutHeapTuple(Relation relation,
Buffer buffer,
HeapTuple tuple,
- bool token)
+ bool token,
+ OffsetNumber root_offnum)
{
Page pageHeader;
OffsetNumber offnum;
@@ -60,17 +68,24 @@ RelationPutHeapTuple(Relation relation,
ItemPointerSet(&(tuple->t_self), BufferGetBlockNumber(buffer), offnum);
/*
- * Insert the correct position into CTID of the stored tuple, too (unless
- * this is a speculative insertion, in which case the token is held in
- * CTID field instead)
+ * Set block number and the root offset into CTID of the stored tuple, too
+ * (unless this is a speculative insertion, in which case the token is held
+ * in CTID field instead).
*/
if (!token)
{
ItemId itemId = PageGetItemId(pageHeader, offnum);
Item item = PageGetItem(pageHeader, itemId);
+ /* Copy t_ctid to set the correct block number. */
((HeapTupleHeader) item)->t_ctid = tuple->t_self;
+
+ if (!OffsetNumberIsValid(root_offnum))
+ root_offnum = offnum;
+ HeapTupleHeaderSetHeapLatest((HeapTupleHeader) item, root_offnum);
}
+
+ return root_offnum;
}
/*
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index d69a266..f54337c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -55,6 +55,8 @@ static void heap_prune_record_redirect(PruneState *prstate,
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
+static void heap_get_root_tuples_internal(Page page,
+ OffsetNumber target_offnum, OffsetNumber *root_offsets);
/*
* Optionally prune and repair fragmentation in the specified page.
@@ -553,6 +555,17 @@ heap_prune_chain(Relation relation, Buffer buffer, OffsetNumber rootoffnum,
if (!HeapTupleHeaderIsHotUpdated(htup))
break;
+
+ /*
+ * If the tuple was HOT-updated and the update was later
+ * aborted, someone could mark this tuple to be the last tuple
+ * in the chain, without clearing the HOT-updated flag. So we must
+ * check if this is the last tuple in the chain and stop following the
+ * CTID, else we risk getting into an infinite recursion (though
+ * prstate->marked[] currently protects against that).
+ */
+ if (HeapTupleHeaderHasRootOffset(htup))
+ break;
/*
* Advance to next chain member.
*/
@@ -726,27 +739,47 @@ heap_page_prune_execute(Buffer buffer,
/*
- * For all items in this page, find their respective root line pointers.
- * If item k is part of a HOT-chain with root at item j, then we set
- * root_offsets[k - 1] = j.
+ * Either for all items in this page or for the given item, find their
+ * respective root line pointers.
+ *
+ * When target_offnum is a valid offset number, the caller is interested in
+ * just one item. In that case, the root line pointer is returned in
+ * root_offsets.
*
- * The passed-in root_offsets array must have MaxHeapTuplesPerPage entries.
- * We zero out all unused entries.
+ * When target_offnum is a InvalidOffsetNumber then the caller wants to know
+ * the root line pointers of all the items in this page. The root_offsets array
+ * must have MaxHeapTuplesPerPage entries in that case. If item k is part of a
+ * HOT-chain with root at item j, then we set root_offsets[k - 1] = j. We zero
+ * out all unused entries.
*
* The function must be called with at least share lock on the buffer, to
* prevent concurrent prune operations.
*
+ * This is not a cheap function since it must scan through all line pointers
+ * and tuples on the page in order to find the root line pointers. To minimize
+ * the cost, we break early if target_offnum is specified and root line pointer
+ * to target_offnum is found.
+ *
* Note: The information collected here is valid only as long as the caller
* holds a pin on the buffer. Once pin is released, a tuple might be pruned
* and reused by a completely unrelated tuple.
+ *
+ * Note: This function must not be called inside a critical section because it
+ * internally calls HeapTupleHeaderGetUpdateXid which somewhere down the stack
+ * may try to allocate heap memory. Memory allocation is disallowed in a
+ * critical section.
*/
-void
-heap_get_root_tuples(Page page, OffsetNumber *root_offsets)
+static void
+heap_get_root_tuples_internal(Page page, OffsetNumber target_offnum,
+ OffsetNumber *root_offsets)
{
OffsetNumber offnum,
maxoff;
- MemSet(root_offsets, 0, MaxHeapTuplesPerPage * sizeof(OffsetNumber));
+ if (OffsetNumberIsValid(target_offnum))
+ *root_offsets = InvalidOffsetNumber;
+ else
+ MemSet(root_offsets, 0, MaxHeapTuplesPerPage * sizeof(OffsetNumber));
maxoff = PageGetMaxOffsetNumber(page);
for (offnum = FirstOffsetNumber; offnum <= maxoff; offnum = OffsetNumberNext(offnum))
@@ -774,9 +807,28 @@ heap_get_root_tuples(Page page, OffsetNumber *root_offsets)
/*
* This is either a plain tuple or the root of a HOT-chain.
- * Remember it in the mapping.
+ *
+ * If the target_offnum is specified and if we found its mapping,
+ * return.
*/
- root_offsets[offnum - 1] = offnum;
+ if (OffsetNumberIsValid(target_offnum))
+ {
+ if (target_offnum == offnum)
+ {
+ root_offsets[0] = offnum;
+ return;
+ }
+ /*
+ * No need to remember mapping for any other item. The
+ * root_offsets array may not even has place for them. So be
+ * careful about not writing past the array.
+ */
+ }
+ else
+ {
+ /* Remember it in the mapping. */
+ root_offsets[offnum - 1] = offnum;
+ }
/* If it's not the start of a HOT-chain, we're done with it */
if (!HeapTupleHeaderIsHotUpdated(htup))
@@ -817,15 +869,65 @@ heap_get_root_tuples(Page page, OffsetNumber *root_offsets)
!TransactionIdEquals(priorXmax, HeapTupleHeaderGetXmin(htup)))
break;
- /* Remember the root line pointer for this item */
- root_offsets[nextoffnum - 1] = offnum;
+ /*
+ * If target_offnum is specified and we found its mapping, return.
+ */
+ if (OffsetNumberIsValid(target_offnum))
+ {
+ if (nextoffnum == target_offnum)
+ {
+ root_offsets[0] = offnum;
+ return;
+ }
+ /*
+ * No need to remember mapping for any other item. The
+ * root_offsets array may not even has place for them. So be
+ * careful about not writing past the array.
+ */
+ }
+ else
+ {
+ /* Remember the root line pointer for this item. */
+ root_offsets[nextoffnum - 1] = offnum;
+ }
/* Advance to next chain member, if any */
if (!HeapTupleHeaderIsHotUpdated(htup))
break;
+ /*
+ * If the tuple was HOT-updated and the update was later aborted,
+ * someone could mark this tuple to be the last tuple in the chain
+ * and store root offset in CTID, without clearing the HOT-updated
+ * flag. So we must check if CTID is actually root offset and break
+ * to avoid infinite recursion.
+ */
+ if (HeapTupleHeaderHasRootOffset(htup))
+ break;
+
nextoffnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
}
}
}
+
+/*
+ * Get root line pointer for the given tuple.
+ */
+OffsetNumber
+heap_get_root_tuple(Page page, OffsetNumber target_offnum)
+{
+ OffsetNumber offnum = InvalidOffsetNumber;
+ heap_get_root_tuples_internal(page, target_offnum, &offnum);
+ return offnum;
+}
+
+/*
+ * Get root line pointers for all tuples in the page
+ */
+void
+heap_get_root_tuples(Page page, OffsetNumber *root_offsets)
+{
+ return heap_get_root_tuples_internal(page, InvalidOffsetNumber,
+ root_offsets);
+}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index d7f65a5..2d3ae9b 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -421,14 +421,18 @@ rewrite_heap_tuple(RewriteState state,
*/
if (!((old_tuple->t_data->t_infomask & HEAP_XMAX_INVALID) ||
HeapTupleHeaderIsOnlyLocked(old_tuple->t_data)) &&
- !(ItemPointerEquals(&(old_tuple->t_self),
- &(old_tuple->t_data->t_ctid))))
+ !(HeapTupleHeaderIsHeapLatest(old_tuple->t_data, &old_tuple->t_self)))
{
OldToNewMapping mapping;
memset(&hashkey, 0, sizeof(hashkey));
hashkey.xmin = HeapTupleHeaderGetUpdateXid(old_tuple->t_data);
- hashkey.tid = old_tuple->t_data->t_ctid;
+
+ /*
+ * We've already checked that this is not the last tuple in the chain,
+ * so fetch the next TID in the chain.
+ */
+ HeapTupleHeaderGetNextTid(old_tuple->t_data, &hashkey.tid);
mapping = (OldToNewMapping)
hash_search(state->rs_old_new_tid_map, &hashkey,
@@ -441,7 +445,7 @@ rewrite_heap_tuple(RewriteState state,
* set the ctid of this tuple to point to the new location, and
* insert it right away.
*/
- new_tuple->t_data->t_ctid = mapping->new_tid;
+ HeapTupleHeaderSetNextTid(new_tuple->t_data, &mapping->new_tid);
/* We don't need the mapping entry anymore */
hash_search(state->rs_old_new_tid_map, &hashkey,
@@ -527,7 +531,7 @@ rewrite_heap_tuple(RewriteState state,
new_tuple = unresolved->tuple;
free_new = true;
old_tid = unresolved->old_tid;
- new_tuple->t_data->t_ctid = new_tid;
+ HeapTupleHeaderSetNextTid(new_tuple->t_data, &new_tid);
/*
* We don't need the hash entry anymore, but don't free its
@@ -733,7 +737,12 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
newitemid = PageGetItemId(page, newoff);
onpage_tup = (HeapTupleHeader) PageGetItem(page, newitemid);
- onpage_tup->t_ctid = tup->t_self;
+ /*
+ * Set t_ctid just to ensure that block number is copied correctly, but
+ * then immediately mark the tuple as the latest.
+ */
+ HeapTupleHeaderSetNextTid(onpage_tup, &tup->t_self);
+ HeapTupleHeaderSetHeapLatest(onpage_tup, newoff);
}
/* If heaptup is a private copy, release it. */
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 5242dee..2142273 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -789,7 +789,8 @@ retry:
DirtySnapshot.speculativeToken &&
TransactionIdPrecedes(GetCurrentTransactionId(), xwait))))
{
- ctid_wait = tup->t_data->t_ctid;
+ if (!HeapTupleHeaderIsHeapLatest(tup->t_data, &tup->t_self))
+ HeapTupleHeaderGetNextTid(tup->t_data, &ctid_wait);
reason_wait = indexInfo->ii_ExclusionOps ?
XLTW_RecheckExclusionConstr : XLTW_InsertIndex;
index_endscan(index_scan);
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index f5cd65d..44a501f 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -2592,7 +2592,7 @@ EvalPlanQualFetch(EState *estate, Relation relation, int lockmode,
* As above, it should be safe to examine xmax and t_ctid without the
* buffer content lock, because they can't be changing.
*/
- if (ItemPointerEquals(&tuple.t_self, &tuple.t_data->t_ctid))
+ if (HeapTupleHeaderIsHeapLatest(tuple.t_data, &tuple.t_self))
{
/* deleted, so forget about it */
ReleaseBuffer(buffer);
@@ -2600,7 +2600,7 @@ EvalPlanQualFetch(EState *estate, Relation relation, int lockmode,
}
/* updated, so look at the updated row */
- tuple.t_self = tuple.t_data->t_ctid;
+ HeapTupleHeaderGetNextTid(tuple.t_data, &tuple.t_self);
/* updated row should have xmin matching this xmax */
priorXmax = HeapTupleHeaderGetUpdateXid(tuple.t_data);
ReleaseBuffer(buffer);
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 7e85510..5540e12 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -190,6 +190,7 @@ extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
OffsetNumber *nowunused, int nunused);
+extern OffsetNumber heap_get_root_tuple(Page page, OffsetNumber target_offnum);
extern void heap_get_root_tuples(Page page, OffsetNumber *root_offsets);
/* in heap/syncscan.c */
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index b285f17..e6019d5 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -193,6 +193,8 @@ typedef struct xl_heap_update
uint8 flags;
TransactionId new_xmax; /* xmax of the new tuple */
OffsetNumber new_offnum; /* new tuple's offset */
+ OffsetNumber root_offnum; /* offset of the root line pointer in case of
+ HOT or WARM update */
/*
* If XLOG_HEAP_CONTAINS_OLD_TUPLE or XLOG_HEAP_CONTAINS_OLD_KEY flags are
@@ -200,7 +202,7 @@ typedef struct xl_heap_update
*/
} xl_heap_update;
-#define SizeOfHeapUpdate (offsetof(xl_heap_update, new_offnum) + sizeof(OffsetNumber))
+#define SizeOfHeapUpdate (offsetof(xl_heap_update, root_offnum) + sizeof(OffsetNumber))
/*
* This is what we need to know about vacuum page cleanup/redirect
diff --git a/src/include/access/hio.h b/src/include/access/hio.h
index 2824f23..921cb37 100644
--- a/src/include/access/hio.h
+++ b/src/include/access/hio.h
@@ -35,8 +35,8 @@ typedef struct BulkInsertStateData
} BulkInsertStateData;
-extern void RelationPutHeapTuple(Relation relation, Buffer buffer,
- HeapTuple tuple, bool token);
+extern OffsetNumber RelationPutHeapTuple(Relation relation, Buffer buffer,
+ HeapTuple tuple, bool token, OffsetNumber root_offnum);
extern Buffer RelationGetBufferForTuple(Relation relation, Size len,
Buffer otherBuffer, int options,
BulkInsertState bistate,
diff --git a/src/include/access/htup_details.h b/src/include/access/htup_details.h
index a6c7e31..7552186 100644
--- a/src/include/access/htup_details.h
+++ b/src/include/access/htup_details.h
@@ -260,13 +260,19 @@ struct HeapTupleHeaderData
* information stored in t_infomask2:
*/
#define HEAP_NATTS_MASK 0x07FF /* 11 bits for number of attributes */
-/* bits 0x1800 are available */
+/* bits 0x0800 are available */
+#define HEAP_LATEST_TUPLE 0x1000 /*
+ * This is the last tuple in chain and
+ * ip_posid points to the root line
+ * pointer
+ */
#define HEAP_KEYS_UPDATED 0x2000 /* tuple was updated and key cols
* modified, or tuple deleted */
#define HEAP_HOT_UPDATED 0x4000 /* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE 0x8000 /* this is heap-only tuple */
-#define HEAP2_XACT_MASK 0xE000 /* visibility-related bits */
+#define HEAP2_XACT_MASK 0xF000 /* visibility-related bits */
+
/*
* HEAP_TUPLE_HAS_MATCH is a temporary flag used during hash joins. It is
@@ -504,6 +510,43 @@ do { \
((tup)->t_infomask2 & HEAP_ONLY_TUPLE) != 0 \
)
+/*
+ * Mark this as the last tuple in the HOT chain. Before PG v10 we used to store
+ * the TID of the tuple itself in t_ctid field to mark the end of the chain.
+ * But starting PG v10, we use a special flag HEAP_LATEST_TUPLE to identify the
+ * last tuple and store the root line pointer of the HOT chain in t_ctid field
+ * instead.
+ *
+ * Note: beware of multiple evaluations of "tup" argument.
+ */
+#define HeapTupleHeaderSetHeapLatest(tup, offnum) \
+do { \
+ AssertMacro(OffsetNumberIsValid(offnum)); \
+ (tup)->t_infomask2 |= HEAP_LATEST_TUPLE; \
+ ItemPointerSetOffsetNumber(&(tup)->t_ctid, (offnum)); \
+} while (0)
+
+#define HeapTupleHeaderClearHeapLatest(tup) \
+( \
+ (tup)->t_infomask2 &= ~HEAP_LATEST_TUPLE \
+)
+
+/*
+ * Starting from PostgreSQL 10, the latest tuple in an update chain has
+ * HEAP_LATEST_TUPLE set; but tuples upgraded from earlier versions do not.
+ * For those, we determine whether a tuple is latest by testing that its t_ctid
+ * points to itself.
+ *
+ * Note: beware of multiple evaluations of "tup" and "tid" arguments.
+ */
+#define HeapTupleHeaderIsHeapLatest(tup, tid) \
+( \
+ (((tup)->t_infomask2 & HEAP_LATEST_TUPLE) != 0) || \
+ ((ItemPointerGetBlockNumber(&(tup)->t_ctid) == ItemPointerGetBlockNumber(tid)) && \
+ (ItemPointerGetOffsetNumber(&(tup)->t_ctid) == ItemPointerGetOffsetNumber(tid))) \
+)
+
+
#define HeapTupleHeaderSetHeapOnly(tup) \
( \
(tup)->t_infomask2 |= HEAP_ONLY_TUPLE \
@@ -542,6 +585,56 @@ do { \
/*
+ * Set the t_ctid chain and also clear the HEAP_LATEST_TUPLE flag since we
+ * now have a new tuple in the chain and this is no longer the last tuple of
+ * the chain.
+ *
+ * Note: beware of multiple evaluations of "tup" argument.
+ */
+#define HeapTupleHeaderSetNextTid(tup, tid) \
+do { \
+ ItemPointerCopy((tid), &((tup)->t_ctid)); \
+ HeapTupleHeaderClearHeapLatest((tup)); \
+} while (0)
+
+/*
+ * Get TID of next tuple in the update chain. Caller must have checked that
+ * we are not already at the end of the chain because in that case t_ctid may
+ * actually store the root line pointer of the HOT chain.
+ *
+ * Note: beware of multiple evaluations of "tup" argument.
+ */
+#define HeapTupleHeaderGetNextTid(tup, next_ctid) \
+do { \
+ AssertMacro(!((tup)->t_infomask2 & HEAP_LATEST_TUPLE)); \
+ ItemPointerCopy(&(tup)->t_ctid, (next_ctid)); \
+} while (0)
+
+/*
+ * Get the root line pointer of the HOT chain. The caller should have confirmed
+ * that the root offset is cached before calling this macro.
+ *
+ * Note: beware of multiple evaluations of "tup" argument.
+ */
+#define HeapTupleHeaderGetRootOffset(tup) \
+( \
+ AssertMacro(((tup)->t_infomask2 & HEAP_LATEST_TUPLE) != 0), \
+ ItemPointerGetOffsetNumber(&(tup)->t_ctid) \
+)
+
+/*
+ * Return whether the tuple has a cached root offset. We don't use
+ * HeapTupleHeaderIsHeapLatest because that one also considers the case of
+ * t_ctid pointing to itself, for tuples migrated from pre v10 clusters. Here
+ * we are only interested in the tuples which are marked with HEAP_LATEST_TUPLE
+ * flag.
+ */
+#define HeapTupleHeaderHasRootOffset(tup) \
+( \
+ ((tup)->t_infomask2 & HEAP_LATEST_TUPLE) != 0 \
+)
+
+/*
* BITMAPLEN(NATTS) -
* Computes size of null bitmap given number of data columns.
*/
[application/octet-stream] 0001_interesting_attrs_v18.patch (11.4K, ../../CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw@mail.gmail.com/6-0001_interesting_attrs_v18.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 8526137..fd4291b 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -96,11 +96,8 @@ static XLogRecPtr log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup,
HeapTuple newtup, HeapTuple old_key_tup,
bool all_visible_cleared, bool new_all_visible_cleared);
-static void HeapSatisfiesHOTandKeyUpdate(Relation relation,
- Bitmapset *hot_attrs,
- Bitmapset *key_attrs, Bitmapset *id_attrs,
- bool *satisfies_hot, bool *satisfies_key,
- bool *satisfies_id,
+static Bitmapset *HeapDetermineModifiedColumns(Relation relation,
+ Bitmapset *interesting_cols,
HeapTuple oldtup, HeapTuple newtup);
static bool heap_acquire_tuplock(Relation relation, ItemPointer tid,
LockTupleMode mode, LockWaitPolicy wait_policy,
@@ -3471,6 +3468,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Bitmapset *hot_attrs;
Bitmapset *key_attrs;
Bitmapset *id_attrs;
+ Bitmapset *interesting_attrs;
+ Bitmapset *modified_attrs;
ItemId lp;
HeapTupleData oldtup;
HeapTuple heaptup;
@@ -3488,9 +3487,6 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
pagefree;
bool have_tuple_lock = false;
bool iscombo;
- bool satisfies_hot;
- bool satisfies_key;
- bool satisfies_id;
bool use_hot_update = false;
bool key_intact;
bool all_visible_cleared = false;
@@ -3517,21 +3513,30 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
errmsg("cannot update tuples during a parallel operation")));
/*
- * Fetch the list of attributes to be checked for HOT update. This is
- * wasted effort if we fail to update or have to put the new tuple on a
- * different page. But we must compute the list before obtaining buffer
- * lock --- in the worst case, if we are doing an update on one of the
- * relevant system catalogs, we could deadlock if we try to fetch the list
- * later. In any case, the relcache caches the data so this is usually
- * pretty cheap.
+ * Fetch the list of attributes to be checked for various operations.
*
- * Note that we get a copy here, so we need not worry about relcache flush
- * happening midway through.
+ * For HOT considerations, this is wasted effort if we fail to update or
+ * have to put the new tuple on a different page. But we must compute the
+ * list before obtaining buffer lock --- in the worst case, if we are doing
+ * an update on one of the relevant system catalogs, we could deadlock if
+ * we try to fetch the list later. In any case, the relcache caches the
+ * data so this is usually pretty cheap.
+ *
+ * We also need columns used by the replica identity, the columns that
+ * are considered the "key" of rows in the table, and columns that are
+ * part of indirect indexes.
+ *
+ * Note that we get copies of each bitmap, so we need not worry about
+ * relcache flush happening midway through.
*/
hot_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_ALL);
key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);
id_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ interesting_attrs = bms_add_members(NULL, hot_attrs);
+ interesting_attrs = bms_add_members(interesting_attrs, key_attrs);
+ interesting_attrs = bms_add_members(interesting_attrs, id_attrs);
+
block = ItemPointerGetBlockNumber(otid);
buffer = ReadBuffer(relation, block);
@@ -3552,7 +3557,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Assert(ItemIdIsNormal(lp));
/*
- * Fill in enough data in oldtup for HeapSatisfiesHOTandKeyUpdate to work
+ * Fill in enough data in oldtup for HeapDetermineModifiedColumns to work
* properly.
*/
oldtup.t_tableOid = RelationGetRelid(relation);
@@ -3578,6 +3583,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Assert(!(newtup->t_data->t_infomask & HEAP_HASOID));
}
+ /* Determine columns modified by the update. */
+ modified_attrs = HeapDetermineModifiedColumns(relation, interesting_attrs,
+ &oldtup, newtup);
+
/*
* If we're not updating any "key" column, we can grab a weaker lock type.
* This allows for more concurrency when we are running simultaneously
@@ -3589,10 +3598,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
* is updates that don't manipulate key columns, not those that
* serendipitiously arrive at the same key values.
*/
- HeapSatisfiesHOTandKeyUpdate(relation, hot_attrs, key_attrs, id_attrs,
- &satisfies_hot, &satisfies_key,
- &satisfies_id, &oldtup, newtup);
- if (satisfies_key)
+ if (!bms_overlap(modified_attrs, key_attrs))
{
*lockmode = LockTupleNoKeyExclusive;
mxact_status = MultiXactStatusNoKeyUpdate;
@@ -3831,6 +3837,8 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(modified_attrs);
+ bms_free(interesting_attrs);
return result;
}
@@ -4135,7 +4143,7 @@ l2:
* to do a HOT update. Check if any of the index columns have been
* changed. If not, then HOT update is possible.
*/
- if (satisfies_hot)
+ if (!bms_overlap(modified_attrs, hot_attrs))
use_hot_update = true;
}
else
@@ -4150,7 +4158,9 @@ l2:
* ExtractReplicaIdentity() will return NULL if nothing needs to be
* logged.
*/
- old_key_tuple = ExtractReplicaIdentity(relation, &oldtup, !satisfies_id, &old_key_copied);
+ old_key_tuple = ExtractReplicaIdentity(relation, &oldtup,
+ bms_overlap(modified_attrs, id_attrs),
+ &old_key_copied);
/* NO EREPORT(ERROR) from here till changes are logged */
START_CRIT_SECTION();
@@ -4298,13 +4308,15 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(modified_attrs);
+ bms_free(interesting_attrs);
return HeapTupleMayBeUpdated;
}
/*
* Check if the specified attribute's value is same in both given tuples.
- * Subroutine for HeapSatisfiesHOTandKeyUpdate.
+ * Subroutine for HeapDetermineModifiedColumns.
*/
static bool
heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
@@ -4338,7 +4350,7 @@ heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
/*
* Extract the corresponding values. XXX this is pretty inefficient if
- * there are many indexed columns. Should HeapSatisfiesHOTandKeyUpdate do
+ * there are many indexed columns. Should HeapDetermineModifiedColumns do
* a single heap_deform_tuple call on each tuple, instead? But that
* doesn't work for system columns ...
*/
@@ -4383,114 +4395,30 @@ heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
/*
* Check which columns are being updated.
*
- * This simultaneously checks conditions for HOT updates, for FOR KEY
- * SHARE updates, and REPLICA IDENTITY concerns. Since much of the time they
- * will be checking very similar sets of columns, and doing the same tests on
- * them, it makes sense to optimize and do them together.
- *
- * We receive three bitmapsets comprising the three sets of columns we're
- * interested in. Note these are destructively modified; that is OK since
- * this is invoked at most once in heap_update.
+ * Given an updated tuple, determine (and return into the output bitmapset),
+ * from those listed as interesting, the set of columns that changed.
*
- * hot_result is set to TRUE if it's okay to do a HOT update (i.e. it does not
- * modified indexed columns); key_result is set to TRUE if the update does not
- * modify columns used in the key; id_result is set to TRUE if the update does
- * not modify columns in any index marked as the REPLICA IDENTITY.
+ * The input bitmapset is destructively modified; that is OK since this is
+ * invoked at most once in heap_update.
*/
-static void
-HeapSatisfiesHOTandKeyUpdate(Relation relation, Bitmapset *hot_attrs,
- Bitmapset *key_attrs, Bitmapset *id_attrs,
- bool *satisfies_hot, bool *satisfies_key,
- bool *satisfies_id,
+static Bitmapset *
+HeapDetermineModifiedColumns(Relation relation, Bitmapset *interesting_cols,
HeapTuple oldtup, HeapTuple newtup)
{
- int next_hot_attnum;
- int next_key_attnum;
- int next_id_attnum;
- bool hot_result = true;
- bool key_result = true;
- bool id_result = true;
-
- /* If REPLICA IDENTITY is set to FULL, id_attrs will be empty. */
- Assert(bms_is_subset(id_attrs, key_attrs));
- Assert(bms_is_subset(key_attrs, hot_attrs));
-
- /*
- * If one of these sets contains no remaining bits, bms_first_member will
- * return -1, and after adding FirstLowInvalidHeapAttributeNumber (which
- * is negative!) we'll get an attribute number that can't possibly be
- * real, and thus won't match any actual attribute number.
- */
- next_hot_attnum = bms_first_member(hot_attrs);
- next_hot_attnum += FirstLowInvalidHeapAttributeNumber;
- next_key_attnum = bms_first_member(key_attrs);
- next_key_attnum += FirstLowInvalidHeapAttributeNumber;
- next_id_attnum = bms_first_member(id_attrs);
- next_id_attnum += FirstLowInvalidHeapAttributeNumber;
+ int attnum;
+ Bitmapset *modified = NULL;
- for (;;)
+ while ((attnum = bms_first_member(interesting_cols)) >= 0)
{
- bool changed;
- int check_now;
-
- /*
- * Since the HOT attributes are a superset of the key attributes and
- * the key attributes are a superset of the id attributes, this logic
- * is guaranteed to identify the next column that needs to be checked.
- */
- if (hot_result && next_hot_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_hot_attnum;
- else if (key_result && next_key_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_key_attnum;
- else if (id_result && next_id_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_id_attnum;
- else
- break;
+ attnum += FirstLowInvalidHeapAttributeNumber;
- /* See whether it changed. */
- changed = !heap_tuple_attr_equals(RelationGetDescr(relation),
- check_now, oldtup, newtup);
- if (changed)
- {
- if (check_now == next_hot_attnum)
- hot_result = false;
- if (check_now == next_key_attnum)
- key_result = false;
- if (check_now == next_id_attnum)
- id_result = false;
-
- /* if all are false now, we can stop checking */
- if (!hot_result && !key_result && !id_result)
- break;
- }
-
- /*
- * Advance the next attribute numbers for the sets that contain the
- * attribute we just checked. As we work our way through the columns,
- * the next_attnum values will rise; but when each set becomes empty,
- * bms_first_member() will return -1 and the attribute number will end
- * up with a value less than FirstLowInvalidHeapAttributeNumber.
- */
- if (hot_result && check_now == next_hot_attnum)
- {
- next_hot_attnum = bms_first_member(hot_attrs);
- next_hot_attnum += FirstLowInvalidHeapAttributeNumber;
- }
- if (key_result && check_now == next_key_attnum)
- {
- next_key_attnum = bms_first_member(key_attrs);
- next_key_attnum += FirstLowInvalidHeapAttributeNumber;
- }
- if (id_result && check_now == next_id_attnum)
- {
- next_id_attnum = bms_first_member(id_attrs);
- next_id_attnum += FirstLowInvalidHeapAttributeNumber;
- }
+ if (!heap_tuple_attr_equals(RelationGetDescr(relation),
+ attnum, oldtup, newtup))
+ modified = bms_add_member(modified,
+ attnum - FirstLowInvalidHeapAttributeNumber);
}
- *satisfies_hot = hot_result;
- *satisfies_key = key_result;
- *satisfies_id = id_result;
+ return modified;
}
/*
[application/octet-stream] 0005_warm_updates_v18.patch (212.5K, ../../CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw@mail.gmail.com/7-0005_warm_updates_v18.patch)
download | inline diff:
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index f2eda67..b356e2b 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -142,6 +142,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git a/contrib/bloom/blvacuum.c b/contrib/bloom/blvacuum.c
index 04abd0f..ff50361 100644
--- a/contrib/bloom/blvacuum.c
+++ b/contrib/bloom/blvacuum.c
@@ -88,7 +88,7 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
while (itup < itupEnd)
{
/* Do we have to delete this tuple? */
- if (callback(&itup->heapPtr, callback_state))
+ if (callback(&itup->heapPtr, false, callback_state) == IBDCR_DELETE)
{
/* Yes; adjust count of tuples that will be left on page */
BloomPageGetOpaque(page)->maxoff--;
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index b22563b..b4a1465 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -116,6 +116,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git a/src/backend/access/gin/ginvacuum.c b/src/backend/access/gin/ginvacuum.c
index c9ccfee..8ed71c5 100644
--- a/src/backend/access/gin/ginvacuum.c
+++ b/src/backend/access/gin/ginvacuum.c
@@ -56,7 +56,8 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
*/
for (i = 0; i < nitem; i++)
{
- if (gvs->callback(items + i, gvs->callback_state))
+ if (gvs->callback(items + i, false, gvs->callback_state) ==
+ IBDCR_DELETE)
{
gvs->result->tuples_removed += 1;
if (!tmpitems)
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 6593771..843389b 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -94,6 +94,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 77d9d12..0955db6 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -202,7 +202,8 @@ gistbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
iid = PageGetItemId(page, i);
idxtuple = (IndexTuple) PageGetItem(page, iid);
- if (callback(&(idxtuple->t_tid), callback_state))
+ if (callback(&(idxtuple->t_tid), false, callback_state) ==
+ IBDCR_DELETE)
todelete[ntodelete++] = i;
else
stats->num_index_tuples += 1;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index cfcec34..2274237 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -75,6 +75,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amwarminsert = hashwarminsert;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
@@ -92,6 +93,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = hashrecheck;
PG_RETURN_POINTER(amroutine);
}
@@ -233,11 +235,11 @@ hashbuildCallback(Relation index,
* Hash on the heap tuple's key, form an index tuple with hash code.
* Find the appropriate location for the new tuple, and put it there.
*/
-bool
-hashinsert(Relation rel, Datum *values, bool *isnull,
+static bool
+hashinsert_internal(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo, bool warm_update)
{
Datum index_values[1];
bool index_isnull[1];
@@ -253,6 +255,11 @@ hashinsert(Relation rel, Datum *values, bool *isnull,
itup = index_form_tuple(RelationGetDescr(rel), index_values, index_isnull);
itup->t_tid = *ht_ctid;
+ if (warm_update)
+ ItemPointerSetFlags(&itup->t_tid, HASH_INDEX_WARM_POINTER);
+ else
+ ItemPointerClearFlags(&itup->t_tid);
+
_hash_doinsert(rel, itup, heapRel);
pfree(itup);
@@ -260,6 +267,26 @@ hashinsert(Relation rel, Datum *values, bool *isnull,
return false;
}
+bool
+hashinsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return hashinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, false);
+}
+
+bool
+hashwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return hashinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, true);
+
+}
/*
* hashgettuple() -- Get the next tuple in the scan.
@@ -274,6 +301,8 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
OffsetNumber offnum;
ItemPointer current;
bool res;
+ IndexTuple itup;
+
/* Hash indexes are always lossy since we store only the hash code */
scan->xs_recheck = true;
@@ -316,8 +345,6 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
offnum <= maxoffnum;
offnum = OffsetNumberNext(offnum))
{
- IndexTuple itup;
-
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
if (ItemPointerEquals(&(so->hashso_heappos), &(itup->t_tid)))
break;
@@ -789,6 +816,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
Page page;
OffsetNumber deletable[MaxOffsetNumber];
int ndeletable = 0;
+ OffsetNumber clearwarm[MaxOffsetNumber];
+ int nclearwarm = 0;
bool retain_pin = false;
vacuum_delay_point();
@@ -806,20 +835,35 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
IndexTuple itup;
Bucket bucket;
bool kill_tuple = false;
+ bool clear_tuple = false;
+ int flags;
+ bool is_warm;
+ IndexBulkDeleteCallbackResult result;
itup = (IndexTuple) PageGetItem(page,
PageGetItemId(page, offno));
htup = &(itup->t_tid);
+ flags = ItemPointerGetFlags(&itup->t_tid);
+ is_warm = ((flags & HASH_INDEX_WARM_POINTER) != 0);
+
/*
* To remove the dead tuples, we strictly want to rely on results
* of callback function. refer btvacuumpage for detailed reason.
*/
- if (callback && callback(htup, callback_state))
+ if (callback)
{
- kill_tuple = true;
- if (tuples_removed)
- *tuples_removed += 1;
+ result = callback(htup, is_warm, callback_state);
+ if (result == IBDCR_DELETE)
+ {
+ kill_tuple = true;
+ if (tuples_removed)
+ *tuples_removed += 1;
+ }
+ else if (result == IBDCR_CLEAR_WARM)
+ {
+ clear_tuple = true;
+ }
}
else if (split_cleanup)
{
@@ -842,6 +886,12 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
}
}
+ if (clear_tuple)
+ {
+ /* clear the WARM pointer */
+ clearwarm[nclearwarm++] = offno;
+ }
+
if (kill_tuple)
{
/* mark the item for deletion */
@@ -866,12 +916,27 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
/*
* Apply deletions, advance to next page and write page if needed.
*/
- if (ndeletable > 0)
+ if (ndeletable > 0 || nclearwarm > 0)
{
/* No ereport(ERROR) until changes are logged */
START_CRIT_SECTION();
- PageIndexMultiDelete(page, deletable, ndeletable);
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the
+ * array and hence offnums recorded earlier won't make any sense
+ * after PageIndexMultiDelete is called.
+ */
+ if (nclearwarm > 0)
+ _hash_clear_items(page, clearwarm, nclearwarm);
+
+ /*
+ * And delete the deletable items
+ */
+ if (ndeletable > 0)
+ PageIndexMultiDelete(page, deletable, ndeletable);
bucket_dirty = true;
/*
@@ -892,6 +957,7 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
XLogRecPtr recptr;
xlrec.is_primary_bucket_page = (buf == bucket_buf) ? true : false;
+ xlrec.nclearitems = nclearwarm;
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, SizeOfHashDelete);
@@ -904,6 +970,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
XLogRegisterBuffer(0, bucket_buf, REGBUF_STANDARD | REGBUF_NO_IMAGE);
XLogRegisterBuffer(1, buf, REGBUF_STANDARD);
+ XLogRegisterBufData(1, (char *) clearwarm,
+ nclearwarm * sizeof(OffsetNumber));
XLogRegisterBufData(1, (char *) deletable,
ndeletable * sizeof(OffsetNumber));
diff --git a/src/backend/access/hash/hash_xlog.c b/src/backend/access/hash/hash_xlog.c
index 8647e8c..fe89ee1 100644
--- a/src/backend/access/hash/hash_xlog.c
+++ b/src/backend/access/hash/hash_xlog.c
@@ -840,6 +840,7 @@ hash_xlog_delete(XLogReaderState *record)
/* replay the record for deleting entries in bucket page */
if (action == BLK_NEEDS_REDO)
{
+ uint16 nclearwarm = xldata->nclearitems;
char *ptr;
Size len;
@@ -849,12 +850,17 @@ hash_xlog_delete(XLogReaderState *record)
if (len > 0)
{
+ OffsetNumber *clearwarm;
OffsetNumber *unused;
OffsetNumber *unend;
- unused = (OffsetNumber *) ptr;
+ clearwarm = (OffsetNumber *) ptr;
+ unused = clearwarm + nclearwarm;
unend = (OffsetNumber *) ((char *) ptr + len);
+ if (nclearwarm)
+ _hash_clear_items(page, clearwarm, nclearwarm);
+
if ((unend - unused) > 0)
PageIndexMultiDelete(page, unused, unend - unused);
}
diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c
index 622cc4b..e689f90 100644
--- a/src/backend/access/hash/hashpage.c
+++ b/src/backend/access/hash/hashpage.c
@@ -1576,3 +1576,17 @@ _hash_getbucketbuf_from_hashkey(Relation rel, uint32 hashkey, int access,
return buf;
}
+
+void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems)
+{
+ int i;
+ IndexTuple itup;
+
+ for (i = 0; i < nclearitems; i++)
+ {
+ itup = (IndexTuple) PageGetItem(page,
+ PageGetItemId(page, clearitemnos[i]));
+ ItemPointerClearFlags(&itup->t_tid);
+ }
+}
diff --git a/src/backend/access/hash/hashsearch.c b/src/backend/access/hash/hashsearch.c
index 2d92049..330ccc5 100644
--- a/src/backend/access/hash/hashsearch.c
+++ b/src/backend/access/hash/hashsearch.c
@@ -59,6 +59,8 @@ _hash_next(IndexScanDesc scan, ScanDirection dir)
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
so->hashso_heappos = itup->t_tid;
+ if (scan->xs_want_itup)
+ scan->xs_itup = itup;
return true;
}
@@ -367,6 +369,9 @@ _hash_first(IndexScanDesc scan, ScanDirection dir)
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
so->hashso_heappos = itup->t_tid;
+ if (scan->xs_want_itup)
+ scan->xs_itup = itup;
+
return true;
}
diff --git a/src/backend/access/hash/hashutil.c b/src/backend/access/hash/hashutil.c
index 2e99719..48464b8 100644
--- a/src/backend/access/hash/hashutil.c
+++ b/src/backend/access/hash/hashutil.c
@@ -17,9 +17,11 @@
#include "access/hash.h"
#include "access/reloptions.h"
#include "access/relscan.h"
+#include "catalog/index.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "storage/buf_internals.h"
+#include "utils/datum.h"
#define CALC_NEW_BUCKET(old_bucket, lowmask) \
old_bucket | (lowmask + 1)
@@ -514,3 +516,70 @@ _hash_kill_items(IndexScanDesc scan)
MarkBufferDirtyHint(so->hashso_curbuf, true);
}
}
+
+/*
+ * Recheck if the heap tuple satisfies the key stored in the index tuple
+ */
+bool
+hashrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
+{
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ bool isavail[INDEX_MAX_KEYS];
+ Datum values2[INDEX_MAX_KEYS];
+ bool isnull2[INDEX_MAX_KEYS];
+ int i;
+ bool equal;
+ int natts = indexRel->rd_rel->relnatts;
+ Form_pg_attribute att;
+
+ FormIndexPlainDatum(indexInfo, heapRel, heapTuple, values, isnull, isavail);
+
+ /*
+ * HASH indexes compute a hash value of the key and store that in the
+ * index. So we must first obtain the hash of the value obtained from the
+ * heap and then do a comparison
+ */
+ _hash_convert_tuple(indexRel, values, isnull, values2, isnull2);
+
+ equal = true;
+ for (i = 1; i <= natts; i++)
+ {
+ Datum indxvalue;
+ bool indxisnull;
+
+ if (!isavail[i - 1])
+ continue;
+
+ indxvalue = index_getattr(indexTuple, i, indexRel->rd_att, &indxisnull);
+
+ /*
+ * If both are NULL then they are equal
+ */
+ if (isnull2[i - 1] && indxisnull)
+ continue;
+
+ /*
+ * If either is NULL then they are not equal
+ */
+ if (isnull2[i - 1] || indxisnull)
+ {
+ equal = false;
+ break;
+ }
+
+ /*
+ * Now do a raw memory comparison
+ */
+ att = indexRel->rd_att->attrs[i - 1];
+ if (!datumIsEqual(values2[i - 1], indxvalue, att->attbyval,
+ att->attlen))
+ {
+ equal = false;
+ break;
+ }
+ }
+
+ return equal;
+}
diff --git a/src/backend/access/heap/README.WARM b/src/backend/access/heap/README.WARM
new file mode 100644
index 0000000..7569227
--- /dev/null
+++ b/src/backend/access/heap/README.WARM
@@ -0,0 +1,305 @@
+src/backend/access/heap/README.WARM
+
+Write Amplification Reduction Method (WARM)
+===========================================
+
+The Heap Only Tuple (HOT) feature greatly eliminated redudant index
+entries and allowed re-use of the dead space occupied by previously
+updated or deleted tuples (see src/backend/access/heap/README.HOT)
+
+One of the necessary conditions for satisfying HOT update is that the
+update must not change a column used in any of the indexes on the table.
+The condition is sometimes hard to meet, especially for complex
+workloads with several indexes on large yet frequently updated tables.
+Worse, sometimes only one or two index columns may be updated, but the
+regular non-HOT update will still insert a new index entry in every
+index on the table, irrespective of whether the key pertaining to the
+index changed or not.
+
+WARM is a technique devised to address these problems.
+
+
+Update Chains With Multiple Index Entries Pointing to the Root
+--------------------------------------------------------------
+
+When a non-HOT update is caused by an index key change, a new index
+entry must be inserted for the changed index. But if the index key
+hasn't changed for other indexes, we don't really need to insert a new
+entry. Even though the existing index entry is pointing to the old
+tuple, the new tuple is reachable via the t_ctid chain. To keep things
+simple, a WARM update requires that the heap block must have enough
+space to store the new version of the tuple. This is same as HOT
+updates.
+
+In WARM, we ensure that every index entry always points to the root of
+the WARM chain. In fact, a WARM chain looks exactly like a HOT chain
+except for the fact that there could be multiple index entries pointing
+to the root of the chain. So when new entry is inserted in an index for
+updated tuple, and if we are doing a WARM update, the new entry is made
+point to the root of the WARM chain.
+
+For example, if we have a table with two columns and two indexes on each
+of the column. When a tuple is first inserted the table, we have exactly
+one index entry pointing to the tuple from both indexes.
+
+ lp [1]
+ [1111, aaaa]
+
+ Index1's entry (1111) points to 1
+ Index2's entry (aaaa) also points to 1
+
+Now if the tuple's second column is updated and if there is room on the
+page, we perform a WARM update. To do so, Index1 does not get any new
+entry and Index2's new entry will still point to the root tuple of the
+chain.
+
+ lp [1] [2]
+ [1111, aaaa]->[111, bbbb]
+
+ Index1's entry (1111) points to 1
+ Index2's old entry (aaaa) points to 1
+ Index2's new entry (bbbb) also points to 1
+
+"A update chain which has more than one index entries pointing to its
+root line pointer is called WARM chain and the action that creates a
+WARM chain is called WARM update."
+
+Since all indexes always point to the root of the WARM chain, even when
+there are more than one index entries, WARM chains can be pruned and
+dead tuples can be removed without a need to do corresponding index
+cleanup.
+
+While this solves the problem of pruning dead tuples from a HOT/WARM
+chain, it also opens up a new technical challenge because now we have a
+situation where a heap tuple is reachable from multiple index entries,
+each having a different index key. While MVCC still ensures that only
+valid tuples are returned, a tuple with a wrong index key may be
+returned because of wrong index entries. In the above example, tuple
+[1111, bbbb] is reachable from both keys (aaaa) as well as (bbbb). For
+this reason, tuples returned from a WARM chain must always be rechecked
+for index key-match.
+
+Recheck Index Key Againt Heap Tuple
+-----------------------------------
+
+Since every Index AM has it's own notion of index tuples, each Index AM
+must implement its own method to recheck heap tuples. For example, a
+hash index stores the hash value of the column and hence recheck routine
+for hash AM must first compute the hash value of the heap attribute and
+then compare it against the value stored in the index tuple.
+
+The patch currently implement recheck routines for hash and btree
+indexes. If the table has an index which doesn't support recheck
+routine, WARM updates are disabled on such tables.
+
+Problem With Duplicate (key, ctid) Index Entries
+------------------------------------------------
+
+The index-key recheck logic works as long as there are no duplicate
+index keys, both pointing to the same WARM chain. In that case, the same
+valid tuple will be reachable via multiple index keys, yet satisfying
+the index key checks. In the above example, if the tuple [1111, bbbb] is
+again updated to [1111, aaaa] and if we insert a new index entry (aaaa)
+pointing to the root line pointer, we will end up with the following
+structure:
+
+ lp [1] [2] [3]
+ [1111, aaaa]->[1111, bbbb]->[1111, aaaa]
+
+ Index1's entry (1111) points to 1
+ Index2's oldest entry (aaaa) points to 1
+ Index2's old entry (bbbb) also points to 1
+ Index2's new entry (aaaa) also points to 1
+
+We must solve this problem to ensure that the same tuple is not
+reachable via multiple index pointers. There are couple of ways to
+address this issue:
+
+1. Do not allow WARM update to a tuple from a WARM chain. This
+guarantees that there can never be duplicate index entries to the same
+root line pointer because we must have checked for old and new index
+keys while doing the first WARM update.
+
+2. Do not allow duplicate (key, ctid) index pointers. In the above
+example, since (aaaa, 1) already exists in the index, we must not insert
+a duplicate index entry.
+
+The patch currently implements 1 i.e. do not do WARM updates to a tuple
+from a WARM chain. HOT updates are fine because they do not add a new
+index entry.
+
+Even with the restriction, this is a significant improvement because the
+number of regular UPDATEs are curtailed down to half.
+
+Expression and Partial Indexes
+------------------------------
+
+Expressions may evaluate to the same value even if the underlying column
+values have changed. A simple example is an index on "lower(col)" which
+will return the same value if the new heap value only differs in the
+case sensitivity. So we can not solely rely on the heap column check to
+decide whether or not to insert a new index entry for expression
+indexes. Similarly, for partial indexes, the predicate expression must
+be evaluated to decide whether or not to cause a new index entry when
+columns referred in the predicate expressions change.
+
+(None of these things are currently implemented and we squarely disallow
+WARM update if a column from expression indexes or predicate has
+changed).
+
+
+Efficiently Finding the Root Line Pointer
+-----------------------------------------
+
+During WARM update, we must be able to find the root line pointer of the
+tuple being updated. It must be noted that the t_ctid field in the heap
+tuple header is usually used to find the next tuple in the update chain.
+But the tuple that we are updating, must be the last tuple in the update
+chain. In such cases, the c_tid field usually points the tuple itself.
+So in theory, we could use the t_ctid to store additional information in
+the last tuple of the update chain, if the information about the tuple
+being the last tuple is stored elsewhere.
+
+We now utilize another bit from t_infomask2 to explicitly identify that
+this is the last tuple in the update chain.
+
+HEAP_LATEST_TUPLE - When this bit is set, the tuple is the last tuple in
+the update chain. The OffsetNumber part of t_ctid points to the root
+line pointer of the chain when HEAP_LATEST_TUPLE flag is set.
+
+If UPDATE operation is aborted, the last tuple in the update chain
+becomes dead. The root line pointer information stored in the tuple
+which remains the last valid tuple in the chain is also lost. In such
+rare cases, the root line pointer must be found in a hard way by
+scanning the entire heap page.
+
+Tracking WARM Chains
+--------------------
+
+When a tuple is WARM updated, the old, the new and every subsequent tuple in
+the chain is marked with a special HEAP_WARM_UPDATED flag. We use the last
+remaining bit in t_infomask2 to store this information.
+
+When a tuple is returned from a WARM chain, the caller must do additional
+checks to ensure that the tuple matches the index key. Even if the tuple
+precedes the WARM update in the chain, it must still be rechecked for the index
+key match (case when old tuple is returned by the new index key). So we must
+follow the update chain everytime to the end to see check if this is a WARM
+chain.
+
+Converting WARM chains back to HOT chains (VACUUM ?)
+----------------------------------------------------
+
+The current implementation of WARM allows only one WARM update per
+chain. This simplifies the design and addresses certain issues around
+duplicate scans. But this also implies that the benefit of WARM will be
+no more than 50%, which is still significant, but if we could return
+WARM chains back to normal status, we could do far more WARM updates.
+
+A distinct property of a WARM chain is that at least one index has more
+than one live index entries pointing to the root of the chain. In other
+words, if we can remove duplicate entry from every index or conclusively
+prove that there are no duplicate index entries for the root line
+pointer, the chain can again be marked as HOT.
+
+Here is one idea:
+
+A WARM chain has two parts, separated by the tuple that caused WARM
+update. All tuples in each part has matching index keys, but certain
+index keys may not match between these two parts. Lets say we mark heap
+tuples in the second part with a special HEAP_WARM_TUPLE flag. Similarly, the
+new index entries caused by the first WARM update are also marked with
+INDEX_WARM_POINTER flags.
+
+There are two distinct parts of the WARM chain. The first part where none of
+the tuples have HEAP_WARM_TUPLE flag set and the second part where every tuple
+has the flag set. Each of these parts satisfy HOT property on its own i.e. all
+tuples have the same value for indexed columns. But these two parts are
+separated by the WARM update which breaks HOT property for one or more indexes.
+
+Heap chain: [1] [2] [3] [4]
+ [aaaa, 1111] -> [aaaa, 1111] -> [bbbb, 1111]W -> [bbbb, 1111]W
+
+Index1: (aaaa) points to 1 (satisfies only tuples without W)
+ (bbbb)W points to 1 (satisfies only tuples marked with W)
+
+Index2: (1111) points to 1 (satisfies tuples with and without W)
+
+
+It's clear that for indexes with both pointers, a heap tuple without
+HEAP_WARM_TUPLE flag will be reachable from the index pointer cleared of
+INDEX_WARM_POINTER flag and that with HEAP_WARM_TUPLE flag will be reachable
+from the pointer with INDEX_WARM_POINTER. But for indexes which did not create
+a new entry, tuples with and without the HEAP_WARM_TUPLE flag will be reachable
+from the original index pointer which doesn't have the INDEX_WARM_POINTER flag.
+(there is no pointer with INDEX_WARM_POINTER in such indexes).
+
+During first heap scan of VACUUM, we look for tuples with HEAP_WARM_UPDATED
+set. If all or none of the live tuples in the chain are marked with
+HEAP_WARM_TUPLE flag, then the chain is a candidate for HOT conversion. We
+remember the root line pointer and whether the tuples in the chain had
+HEAP_WARM_TUPLE flags set or not.
+
+If we have a WARM chain with HEAP_WARM_TUPLE set, then our goal is to remove
+the index pointers without INDEX_WARM_POINTER flags and vice versa. But there
+is a catch. For Index2 above, there is only one pointer and it does not have
+the INDEX_WARM_POINTER flag set. Since all heap tuples are reachable only via
+this pointer, it must not be removed. IOW we should remove index pointer
+without INDEX_WARM_POINTER iff a another index pointer with INDEX_WARM_POINTER
+exists. Since index vacuum may visit these pointers in any order, we will need
+another index pass to remove dead index pointers. So in the first index pass we
+check which WARM candidates have 2 index pointers. In the second pass, we
+remove the dead pointer and clear the INDEX_WARM_POINTER flag if that's the
+surviving index pointer.
+
+During the second heap scan, we fix WARM chain by clearing HEAP_WARM_UPDATED
+flag and also clear HEAP_WARM_TUPLE flags on tuples.
+
+There are some more problems around aborted vacuums. For example, if vacuum
+aborts after clearing INDEX_WARM_POINTER flag but before removing the other
+index pointer, we will end up with two index pointers and none of those will
+have INDEX_WARM_POINTER set. But since the HEAP_WARM_UPDATED flag on the heap
+tuple is still set, further WARM updates to the chain will be blocked. I guess
+we will need some special handling for case with multiple index pointers where
+none of the index pointers has INDEX_WARM_POINTER flag set. We can either leave
+these WARM chains alone and let them die with a subsequent non-WARM update or
+must apply heap-recheck logic during index vacuum to find the dead pointer.
+Given that vacuum-aborts are not common, I am inclined to leave this case
+unhandled. We must still check for presence of multiple index pointers without
+INDEX_WARM_POINTER flags and ensure that we don't accidently remove either of
+these pointers and also must not clear WARM chains.
+
+CREATE INDEX CONCURRENTLY
+-------------------------
+
+Currently CREATE INDEX CONCURRENTLY (CIC) is implemented as a 3-phase
+process. In the first phase, we create catalog entry for the new index
+so that the index is visible to all other backends, but still don't use
+it for either read or write. But we ensure that no new broken HOT
+chains are created by new transactions. In the second phase, we build
+the new index using a MVCC snapshot and then make the index available
+for inserts. We then do another pass over the index and insert any
+missing tuples, everytime indexing only it's root line pointer. See
+README.HOT for details about how HOT impacts CIC and how various
+challenges are tackeled.
+
+WARM poses another challenge because it allows creation of HOT chains
+even when an index key is changed. But since the index is not ready for
+insertion until the second phase is over, we might end up with a
+situation where the HOT chain has tuples with different index columns,
+yet only one of these values are indexed by the new index. Note that
+during the third phase, we only index tuples whose root line pointer is
+missing from the index. But we can't easily check if the existing index
+tuple is actually indexing the heap tuple visible to the new MVCC
+snapshot. Finding that information will require us to query the index
+again for every tuple in the chain, especially if it's a WARM tuple.
+This would require repeated access to the index. Another option would be
+to return index keys along with the heap TIDs when index is scanned for
+collecting all indexed TIDs during third phase. We can then compare the
+heap tuple against the already indexed key and decide whether or not to
+index the new tuple.
+
+We solve this problem more simply by disallowing WARM updates until the
+index is ready for insertion. We don't need to disallow WARM on a
+wholesale basis, but only those updates that change the columns of the
+new index are disallowed to be WARM updates.
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 26a7af4..c86fbc6 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1974,6 +1974,206 @@ heap_fetch(Relation relation,
}
/*
+ * Check status of a (possibly) WARM chain.
+ *
+ * This function looks at a HOT/WARM chain starting at tid and return a bitmask
+ * of information. We only follow the chain as long as it's known to be valid
+ * HOT chain. Information returned by the function consists of:
+ *
+ * HCWC_WARM_UPDATED_TUPLE - a tuple with HEAP_WARM_UPDATED is found somewhere
+ * in the chain. Note that when a tuple is WARM
+ * updated, both old and new versions are marked
+ * with this flag/
+ *
+ * HCWC_WARM_TUPLE - a tuple with HEAP_WARM_TUPLE is found somewhere in
+ * the chain.
+ *
+ * HCWC_CLEAR_TUPLE - a tuple without HEAP_WARM_TUPLE is found somewhere in
+ * the chain.
+ *
+ * If stop_at_warm is true, we stop when the first HEAP_WARM_UPDATED tuple is
+ * found and return information collected so far.
+ */
+HeapCheckWarmChainStatus
+heap_check_warm_chain(Page dp, ItemPointer tid, bool stop_at_warm)
+{
+ TransactionId prev_xmax = InvalidTransactionId;
+ OffsetNumber offnum;
+ HeapTupleData heapTuple;
+ HeapCheckWarmChainStatus status = 0;
+
+ offnum = ItemPointerGetOffsetNumber(tid);
+ heapTuple.t_self = *tid;
+ /* Scan through possible multiple members of HOT-chain */
+ for (;;)
+ {
+ ItemId lp;
+
+ /* check for bogus TID */
+ if (offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(dp))
+ break;
+
+ lp = PageGetItemId(dp, offnum);
+
+ /* check for unused, dead, or redirected items */
+ if (!ItemIdIsNormal(lp))
+ {
+ if (ItemIdIsRedirected(lp))
+ {
+ /* Follow the redirect */
+ offnum = ItemIdGetRedirect(lp);
+ continue;
+ }
+ /* else must be end of chain */
+ break;
+ }
+
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(dp, lp);
+ ItemPointerSetOffsetNumber(&heapTuple.t_self, offnum);
+
+ /*
+ * The xmin should match the previous xmax value, else chain is
+ * broken.
+ */
+ if (TransactionIdIsValid(prev_xmax) &&
+ !TransactionIdEquals(prev_xmax,
+ HeapTupleHeaderGetXmin(heapTuple.t_data)))
+ break;
+
+
+ if (HeapTupleHeaderIsWarmUpdated(heapTuple.t_data))
+ {
+ /* We found a WARM_UPDATED tuple */
+ status |= HCWC_WARM_UPDATED_TUPLE;
+
+ /*
+ * If we've been told to stop at the first WARM_UPDATED tuple, just
+ * return whatever information collected so far.
+ */
+ if (stop_at_warm)
+ return status;
+
+ /*
+ * Remember whether it's a CLEAR or a WARM tuple.
+ */
+ if (HeapTupleHeaderIsWarm(heapTuple.t_data))
+ status |= HCWC_WARM_TUPLE;
+ else
+ status |= HCWC_CLEAR_TUPLE;
+ }
+ else
+ /* Must be a regular, non-WARM tuple */
+ status |= HCWC_CLEAR_TUPLE;
+
+ /*
+ * Check to see if HOT chain continues past this tuple; if so fetch
+ * the next offnum and loop around.
+ */
+ if (!HeapTupleIsHotUpdated(&heapTuple))
+ break;
+
+ /*
+ * It can't be a HOT chain if the tuple contains root line pointer
+ */
+ if (HeapTupleHeaderHasRootOffset(heapTuple.t_data))
+ break;
+
+ offnum = ItemPointerGetOffsetNumber(&heapTuple.t_data->t_ctid);
+ prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple.t_data);
+ }
+
+ /* All OK. No need to recheck */
+ return status;
+}
+
+/*
+ * Scan through the WARM chain starting at tid and reset all WARM related
+ * flags. At the end, the chain will have all characteristics of a regular HOT
+ * chain.
+ *
+ * Return the number of cleared offnums. Cleared offnums are returned in the
+ * passed-in cleared_offnums array. The caller must ensure that the array is
+ * large enough to hold maximum offnums that can be cleared by this invokation
+ * of heap_clear_warm_chain().
+ */
+int
+heap_clear_warm_chain(Page dp, ItemPointer tid, OffsetNumber *cleared_offnums)
+{
+ TransactionId prev_xmax = InvalidTransactionId;
+ OffsetNumber offnum;
+ HeapTupleData heapTuple;
+ int num_cleared = 0;
+
+ offnum = ItemPointerGetOffsetNumber(tid);
+ heapTuple.t_self = *tid;
+ /* Scan through possible multiple members of HOT-chain */
+ for (;;)
+ {
+ ItemId lp;
+
+ /* check for bogus TID */
+ if (offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(dp))
+ break;
+
+ lp = PageGetItemId(dp, offnum);
+
+ /* check for unused, dead, or redirected items */
+ if (!ItemIdIsNormal(lp))
+ {
+ if (ItemIdIsRedirected(lp))
+ {
+ /* Follow the redirect */
+ offnum = ItemIdGetRedirect(lp);
+ continue;
+ }
+ /* else must be end of chain */
+ break;
+ }
+
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(dp, lp);
+ ItemPointerSetOffsetNumber(&heapTuple.t_self, offnum);
+
+ /*
+ * The xmin should match the previous xmax value, else chain is
+ * broken.
+ */
+ if (TransactionIdIsValid(prev_xmax) &&
+ !TransactionIdEquals(prev_xmax,
+ HeapTupleHeaderGetXmin(heapTuple.t_data)))
+ break;
+
+
+ /*
+ * Clear WARM_UPDATED and WARM flags.
+ */
+ if (HeapTupleHeaderIsWarmUpdated(heapTuple.t_data))
+ {
+ HeapTupleHeaderClearWarmUpdated(heapTuple.t_data);
+ HeapTupleHeaderClearWarm(heapTuple.t_data);
+ cleared_offnums[num_cleared++] = offnum;
+ }
+
+ /*
+ * Check to see if HOT chain continues past this tuple; if so fetch
+ * the next offnum and loop around.
+ */
+ if (!HeapTupleIsHotUpdated(&heapTuple))
+ break;
+
+ /*
+ * It can't be a HOT chain if the tuple contains root line pointer
+ */
+ if (HeapTupleHeaderHasRootOffset(heapTuple.t_data))
+ break;
+
+ offnum = ItemPointerGetOffsetNumber(&heapTuple.t_data->t_ctid);
+ prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple.t_data);
+ }
+
+ return num_cleared;
+}
+
+/*
* heap_hot_search_buffer - search HOT chain for tuple satisfying snapshot
*
* On entry, *tid is the TID of a tuple (either a simple tuple, or the root
@@ -1993,11 +2193,14 @@ heap_fetch(Relation relation,
* Unlike heap_fetch, the caller must already have pin and (at least) share
* lock on the buffer; it is still pinned/locked at exit. Also unlike
* heap_fetch, we do not report any pgstats count; caller may do so if wanted.
+ *
+ * recheck should be set false on entry by caller, will be set true on exit
+ * if a WARM tuple is encountered.
*/
bool
heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
Snapshot snapshot, HeapTuple heapTuple,
- bool *all_dead, bool first_call)
+ bool *all_dead, bool first_call, bool *recheck)
{
Page dp = (Page) BufferGetPage(buffer);
TransactionId prev_xmax = InvalidTransactionId;
@@ -2051,9 +2254,12 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
ItemPointerSetOffsetNumber(&heapTuple->t_self, offnum);
/*
- * Shouldn't see a HEAP_ONLY tuple at chain start.
+ * Shouldn't see a HEAP_ONLY tuple at chain start, unless we are
+ * dealing with a WARM updated tuple in which case deferred triggers
+ * may request to fetch a WARM tuple from middle of a chain.
*/
- if (at_chain_start && HeapTupleIsHeapOnly(heapTuple))
+ if (at_chain_start && HeapTupleIsHeapOnly(heapTuple) &&
+ !HeapTupleIsWarmUpdated(heapTuple))
break;
/*
@@ -2066,6 +2272,20 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
break;
/*
+ * Check if there exists a WARM tuple somewhere down the chain and set
+ * recheck to TRUE.
+ *
+ * XXX This is not very efficient right now, and we should look for
+ * possible improvements here.
+ */
+ if (recheck && *recheck == false)
+ {
+ HeapCheckWarmChainStatus status;
+ status = heap_check_warm_chain(dp, &heapTuple->t_self, true);
+ *recheck = HCWC_IS_WARM_UPDATED(status);
+ }
+
+ /*
* When first_call is true (and thus, skip is initially false) we'll
* return the first tuple we find. But on later passes, heapTuple
* will initially be pointing to the tuple we returned last time.
@@ -2114,7 +2334,8 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
* Check to see if HOT chain continues past this tuple; if so fetch
* the next offnum and loop around.
*/
- if (HeapTupleIsHotUpdated(heapTuple))
+ if (HeapTupleIsHotUpdated(heapTuple) &&
+ !HeapTupleHeaderHasRootOffset(heapTuple->t_data))
{
Assert(ItemPointerGetBlockNumber(&heapTuple->t_data->t_ctid) ==
ItemPointerGetBlockNumber(tid));
@@ -2138,18 +2359,41 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
*/
bool
heap_hot_search(ItemPointer tid, Relation relation, Snapshot snapshot,
- bool *all_dead)
+ bool *all_dead, bool *recheck, Buffer *cbuffer,
+ HeapTuple heapTuple)
{
bool result;
Buffer buffer;
- HeapTupleData heapTuple;
+ ItemPointerData ret_tid = *tid;
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_SHARE);
- result = heap_hot_search_buffer(tid, relation, buffer, snapshot,
- &heapTuple, all_dead, true);
- LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
- ReleaseBuffer(buffer);
+ result = heap_hot_search_buffer(&ret_tid, relation, buffer, snapshot,
+ heapTuple, all_dead, true, recheck);
+
+ /*
+ * If we are returning a potential candidate tuple from this chain and the
+ * caller has requested for "recheck" hint, keep the buffer locked and
+ * pinned. The caller must release the lock and pin on the buffer in all
+ * such cases.
+ */
+ if (!result || !recheck || !(*recheck))
+ {
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buffer);
+ }
+
+ /*
+ * Set the caller supplied tid with the actual location of the tuple being
+ * returned.
+ */
+ if (result)
+ {
+ *tid = ret_tid;
+ if (cbuffer)
+ *cbuffer = buffer;
+ }
+
return result;
}
@@ -2792,7 +3036,7 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
{
XLogRecPtr recptr;
xl_heap_multi_insert *xlrec;
- uint8 info = XLOG_HEAP2_MULTI_INSERT;
+ uint8 info = XLOG_HEAP_MULTI_INSERT;
char *tupledata;
int totaldatalen;
char *scratchptr = scratch;
@@ -2889,7 +3133,7 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP2_ID, info);
+ recptr = XLogInsert(RM_HEAP_ID, info);
PageSetLSN(page, recptr);
}
@@ -3313,7 +3557,9 @@ l1:
}
/* store transaction information of xact deleting the tuple */
- tp.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ tp.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(tp.t_data))
+ tp.t_data->t_infomask &= ~HEAP_MOVED;
tp.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
tp.t_data->t_infomask |= new_infomask;
tp.t_data->t_infomask2 |= new_infomask2;
@@ -3508,15 +3754,18 @@ simple_heap_delete(Relation relation, ItemPointer tid)
HTSU_Result
heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
CommandId cid, Snapshot crosscheck, bool wait,
- HeapUpdateFailureData *hufd, LockTupleMode *lockmode)
+ HeapUpdateFailureData *hufd, LockTupleMode *lockmode,
+ Bitmapset **modified_attrsp, bool *warm_update)
{
HTSU_Result result;
TransactionId xid = GetCurrentTransactionId();
Bitmapset *hot_attrs;
Bitmapset *key_attrs;
Bitmapset *id_attrs;
+ Bitmapset *exprindx_attrs;
Bitmapset *interesting_attrs;
Bitmapset *modified_attrs;
+ Bitmapset *notready_attrs;
ItemId lp;
HeapTupleData oldtup;
HeapTuple heaptup;
@@ -3537,6 +3786,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
bool have_tuple_lock = false;
bool iscombo;
bool use_hot_update = false;
+ bool use_warm_update = false;
bool key_intact;
bool all_visible_cleared = false;
bool all_visible_cleared_new = false;
@@ -3561,6 +3811,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
(errcode(ERRCODE_INVALID_TRANSACTION_STATE),
errmsg("cannot update tuples during a parallel operation")));
+ /* Assume no-warm update */
+ if (warm_update)
+ *warm_update = false;
+
/*
* Fetch the list of attributes to be checked for various operations.
*
@@ -3582,10 +3836,17 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);
id_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ exprindx_attrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_EXPR_PREDICATE);
+ notready_attrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_NOTREADY);
+
+
interesting_attrs = bms_add_members(NULL, hot_attrs);
interesting_attrs = bms_add_members(interesting_attrs, key_attrs);
interesting_attrs = bms_add_members(interesting_attrs, id_attrs);
-
+ interesting_attrs = bms_add_members(interesting_attrs, exprindx_attrs);
+ interesting_attrs = bms_add_members(interesting_attrs, notready_attrs);
block = ItemPointerGetBlockNumber(otid);
offnum = ItemPointerGetOffsetNumber(otid);
@@ -3637,6 +3898,9 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
modified_attrs = HeapDetermineModifiedColumns(relation, interesting_attrs,
&oldtup, newtup);
+ if (modified_attrsp)
+ *modified_attrsp = bms_copy(modified_attrs);
+
/*
* If we're not updating any "key" column, we can grab a weaker lock type.
* This allows for more concurrency when we are running simultaneously
@@ -3892,6 +4156,7 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(exprindx_attrs);
bms_free(modified_attrs);
bms_free(interesting_attrs);
return result;
@@ -4057,7 +4322,9 @@ l2:
START_CRIT_SECTION();
/* Clear obsolete visibility flags ... */
- oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ oldtup.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(oldtup.t_data))
+ oldtup.t_data->t_infomask &= ~HEAP_MOVED;
oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
HeapTupleClearHotUpdated(&oldtup);
/* ... and store info about transaction updating this tuple */
@@ -4210,6 +4477,24 @@ l2:
*/
if (!bms_overlap(modified_attrs, hot_attrs))
use_hot_update = true;
+ else
+ {
+ /*
+ * If no WARM updates yet on this chain, let this update be a WARM
+ * update.
+ *
+ * We check for both warm and warm updated tuples since if the
+ * previous WARM update aborted, we may still have added
+ * another index entry for this HOT chain. In such situations, we
+ * must not attempt a WARM update.
+ */
+ if (relation->rd_supportswarm &&
+ !bms_overlap(modified_attrs, exprindx_attrs) &&
+ !bms_is_subset(hot_attrs, modified_attrs) &&
+ !bms_overlap(notready_attrs, modified_attrs) &&
+ !HeapTupleIsWarmUpdated(&oldtup))
+ use_warm_update = true;
+ }
}
else
{
@@ -4256,6 +4541,32 @@ l2:
HeapTupleSetHeapOnly(heaptup);
/* Mark the caller's copy too, in case different from heaptup */
HeapTupleSetHeapOnly(newtup);
+
+ /*
+ * Even if we are doing a HOT update, we must carry forward the WARM
+ * flag because we may have already inserted another index entry
+ * pointing to our root and a third entry may create duplicates.
+ *
+ * Note: If we ever have a mechanism to avoid duplicate <key, TID> in
+ * indexes, we could look at relaxing this restriction and allow even
+ * more WARM udpates.
+ */
+ if (HeapTupleIsWarmUpdated(&oldtup))
+ {
+ HeapTupleSetWarmUpdated(heaptup);
+ HeapTupleSetWarmUpdated(newtup);
+ }
+
+ /*
+ * If the old tuple is a WARM tuple then mark the new tuple as a WARM
+ * tuple as well.
+ */
+ if (HeapTupleIsWarm(&oldtup))
+ {
+ HeapTupleSetWarm(heaptup);
+ HeapTupleSetWarm(newtup);
+ }
+
/*
* For HOT (or WARM) updated tuples, we store the offset of the root
* line pointer of this chain in the ip_posid field of the new tuple.
@@ -4268,12 +4579,45 @@ l2:
if (HeapTupleHeaderHasRootOffset(oldtup.t_data))
root_offnum = HeapTupleHeaderGetRootOffset(oldtup.t_data);
}
+ else if (use_warm_update)
+ {
+ /* Mark the old tuple as HOT-updated */
+ HeapTupleSetHotUpdated(&oldtup);
+ HeapTupleSetWarmUpdated(&oldtup);
+
+ /* And mark the new tuple as heap-only */
+ HeapTupleSetHeapOnly(heaptup);
+ /* Mark the new tuple as WARM tuple */
+ HeapTupleSetWarmUpdated(heaptup);
+ /* This update also starts the WARM chain */
+ HeapTupleSetWarm(heaptup);
+ Assert(!HeapTupleIsWarm(&oldtup));
+
+ /* Mark the caller's copy too, in case different from heaptup */
+ HeapTupleSetHeapOnly(newtup);
+ HeapTupleSetWarmUpdated(newtup);
+ HeapTupleSetWarm(newtup);
+
+ if (HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ root_offnum = HeapTupleHeaderGetRootOffset(oldtup.t_data);
+ else
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&(oldtup.t_self)));
+
+ /* Let the caller know we did a WARM update */
+ if (warm_update)
+ *warm_update = true;
+ }
else
{
/* Make sure tuples are correctly marked as not-HOT */
HeapTupleClearHotUpdated(&oldtup);
HeapTupleClearHeapOnly(heaptup);
HeapTupleClearHeapOnly(newtup);
+ HeapTupleClearWarmUpdated(heaptup);
+ HeapTupleClearWarmUpdated(newtup);
+ HeapTupleClearWarm(heaptup);
+ HeapTupleClearWarm(newtup);
root_offnum = InvalidOffsetNumber;
}
@@ -4292,7 +4636,9 @@ l2:
HeapTupleHeaderSetHeapLatest(newtup->t_data, root_offnum);
/* Clear obsolete visibility flags, possibly set by ourselves above... */
- oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ oldtup.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(oldtup.t_data))
+ oldtup.t_data->t_infomask &= ~HEAP_MOVED;
oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
/* ... and store info about transaction updating this tuple */
Assert(TransactionIdIsValid(xmax_old_tuple));
@@ -4383,7 +4729,10 @@ l2:
if (have_tuple_lock)
UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode);
- pgstat_count_heap_update(relation, use_hot_update);
+ /*
+ * Count HOT and WARM updates separately
+ */
+ pgstat_count_heap_update(relation, use_hot_update, use_warm_update);
/*
* If heaptup is a private copy, release it. Don't forget to copy t_self
@@ -4523,7 +4872,8 @@ HeapDetermineModifiedColumns(Relation relation, Bitmapset *interesting_cols,
* via ereport().
*/
void
-simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup)
+simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup,
+ Bitmapset **modified_attrs, bool *warm_update)
{
HTSU_Result result;
HeapUpdateFailureData hufd;
@@ -4532,7 +4882,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup)
result = heap_update(relation, otid, tup,
GetCurrentCommandId(true), InvalidSnapshot,
true /* wait for commit */ ,
- &hufd, &lockmode);
+ &hufd, &lockmode, modified_attrs, warm_update);
switch (result)
{
case HeapTupleSelfUpdated:
@@ -6209,7 +6559,9 @@ heap_abort_speculative(Relation relation, HeapTuple tuple)
PageSetPrunable(page, RecentGlobalXmin);
/* store transaction information of xact deleting the tuple */
- tp.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ tp.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(tp.t_data))
+ tp.t_data->t_infomask &= ~HEAP_MOVED;
tp.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
/*
@@ -6783,7 +7135,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple, TransactionId cutoff_xid,
* Old-style VACUUM FULL is gone, but we have to keep this code as long as
* we support having MOVED_OFF/MOVED_IN tuples in the database.
*/
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
@@ -6802,7 +7154,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple, TransactionId cutoff_xid,
* have failed; whereas a non-dead MOVED_IN tuple must mean the
* xvac transaction succeeded.
*/
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
frz->frzflags |= XLH_INVALID_XVAC;
else
frz->frzflags |= XLH_FREEZE_XVAC;
@@ -7272,7 +7624,7 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
return true;
}
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
if (TransactionIdIsNormal(xid))
@@ -7355,7 +7707,7 @@ heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
return true;
}
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
if (TransactionIdIsNormal(xid) &&
@@ -7381,7 +7733,7 @@ HeapTupleHeaderAdvanceLatestRemovedXid(HeapTupleHeader tuple,
TransactionId xmax = HeapTupleHeaderGetUpdateXid(tuple);
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
if (TransactionIdPrecedes(*latestRemovedXid, xvac))
*latestRemovedXid = xvac;
@@ -7430,6 +7782,36 @@ log_heap_cleanup_info(RelFileNode rnode, TransactionId latestRemovedXid)
}
/*
+ * Perform XLogInsert for a heap-warm-clear operation. Caller must already
+ * have modified the buffer and marked it dirty.
+ */
+XLogRecPtr
+log_heap_warmclear(Relation reln, Buffer buffer,
+ OffsetNumber *cleared, int ncleared)
+{
+ xl_heap_warmclear xlrec;
+ XLogRecPtr recptr;
+
+ /* Caller should not call me on a non-WAL-logged relation */
+ Assert(RelationNeedsWAL(reln));
+
+ xlrec.ncleared = ncleared;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfHeapWarmClear);
+
+ XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
+
+ if (ncleared > 0)
+ XLogRegisterBufData(0, (char *) cleared,
+ ncleared * sizeof(OffsetNumber));
+
+ recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_WARMCLEAR);
+
+ return recptr;
+}
+
+/*
* Perform XLogInsert for a heap-clean operation. Caller must already
* have modified the buffer and marked it dirty.
*
@@ -7584,6 +7966,7 @@ log_heap_update(Relation reln, Buffer oldbuf,
bool need_tuple_data = RelationIsLogicallyLogged(reln);
bool init;
int bufflags;
+ bool warm_update = false;
/* Caller should not call me on a non-WAL-logged relation */
Assert(RelationNeedsWAL(reln));
@@ -7595,6 +7978,9 @@ log_heap_update(Relation reln, Buffer oldbuf,
else
info = XLOG_HEAP_UPDATE;
+ if (HeapTupleIsWarmUpdated(newtup))
+ warm_update = true;
+
/*
* If the old and new tuple are on the same page, we only need to log the
* parts of the new tuple that were changed. That saves on the amount of
@@ -7668,6 +8054,8 @@ log_heap_update(Relation reln, Buffer oldbuf,
xlrec.flags |= XLH_UPDATE_CONTAINS_OLD_KEY;
}
}
+ if (warm_update)
+ xlrec.flags |= XLH_UPDATE_WARM_UPDATE;
/* If new tuple is the single and first tuple on page... */
if (ItemPointerGetOffsetNumber(&(newtup->t_self)) == FirstOffsetNumber &&
@@ -8082,6 +8470,60 @@ heap_xlog_clean(XLogReaderState *record)
XLogRecordPageWithFreeSpace(rnode, blkno, freespace);
}
+
+/*
+ * Handles HEAP2_WARMCLEAR record type
+ */
+static void
+heap_xlog_warmclear(XLogReaderState *record)
+{
+ XLogRecPtr lsn = record->EndRecPtr;
+ xl_heap_warmclear *xlrec = (xl_heap_warmclear *) XLogRecGetData(record);
+ Buffer buffer;
+ RelFileNode rnode;
+ BlockNumber blkno;
+ XLogRedoAction action;
+
+ XLogRecGetBlockTag(record, 0, &rnode, NULL, &blkno);
+
+ /*
+ * If we have a full-page image, restore it (using a cleanup lock) and
+ * we're done.
+ */
+ action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, true,
+ &buffer);
+ if (action == BLK_NEEDS_REDO)
+ {
+ Page page = (Page) BufferGetPage(buffer);
+ OffsetNumber *cleared;
+ int ncleared;
+ Size datalen;
+ int i;
+
+ cleared = (OffsetNumber *) XLogRecGetBlockData(record, 0, &datalen);
+
+ ncleared = xlrec->ncleared;
+
+ for (i = 0; i < ncleared; i++)
+ {
+ ItemId lp;
+ OffsetNumber offnum = cleared[i];
+ HeapTupleData heapTuple;
+
+ lp = PageGetItemId(page, offnum);
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(page, lp);
+
+ HeapTupleHeaderClearWarmUpdated(heapTuple.t_data);
+ HeapTupleHeaderClearWarm(heapTuple.t_data);
+ }
+
+ PageSetLSN(page, lsn);
+ MarkBufferDirty(buffer);
+ }
+ if (BufferIsValid(buffer))
+ UnlockReleaseBuffer(buffer);
+}
+
/*
* Replay XLOG_HEAP2_VISIBLE record.
*
@@ -8328,7 +8770,9 @@ heap_xlog_delete(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
HeapTupleHeaderClearHotUpdated(htup);
fix_infomask_from_infobits(xlrec->infobits_set,
@@ -8349,7 +8793,7 @@ heap_xlog_delete(XLogReaderState *record)
if (!HeapTupleHeaderHasRootOffset(htup))
{
OffsetNumber root_offnum;
- root_offnum = heap_get_root_tuple(page, xlrec->offnum);
+ root_offnum = heap_get_root_tuple(page, xlrec->offnum);
HeapTupleHeaderSetHeapLatest(htup, root_offnum);
}
@@ -8645,16 +9089,22 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
Size freespace = 0;
XLogRedoAction oldaction;
XLogRedoAction newaction;
+ bool warm_update = false;
/* initialize to keep the compiler quiet */
oldtup.t_data = NULL;
oldtup.t_len = 0;
+ if (xlrec->flags & XLH_UPDATE_WARM_UPDATE)
+ warm_update = true;
+
XLogRecGetBlockTag(record, 0, &rnode, NULL, &newblk);
if (XLogRecGetBlockTag(record, 1, NULL, NULL, &oldblk))
{
/* HOT updates are never done across pages */
Assert(!hot_update);
+ /* WARM updates are never done across pages */
+ Assert(!warm_update);
}
else
oldblk = newblk;
@@ -8714,6 +9164,11 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
&htup->t_infomask2);
HeapTupleHeaderSetXmax(htup, xlrec->old_xmax);
HeapTupleHeaderSetCmax(htup, FirstCommandId, false);
+
+ /* Mark the old tuple has a WARM tuple */
+ if (warm_update)
+ HeapTupleHeaderSetWarmUpdated(htup);
+
/* Set forward chain link in t_ctid */
HeapTupleHeaderSetNextTid(htup, &newtid);
@@ -8849,6 +9304,10 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
HeapTupleHeaderSetCmin(htup, FirstCommandId);
HeapTupleHeaderSetXmax(htup, xlrec->new_xmax);
+ /* Mark the new tuple has a WARM tuple */
+ if (warm_update)
+ HeapTupleHeaderSetWarmUpdated(htup);
+
offnum = PageAddItem(page, (Item) htup, newlen, offnum, true, true);
if (offnum == InvalidOffsetNumber)
elog(PANIC, "failed to add tuple");
@@ -8976,7 +9435,9 @@ heap_xlog_lock(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
fix_infomask_from_infobits(xlrec->infobits_set, &htup->t_infomask,
&htup->t_infomask2);
@@ -9055,7 +9516,9 @@ heap_xlog_lock_updated(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
fix_infomask_from_infobits(xlrec->infobits_set, &htup->t_infomask,
&htup->t_infomask2);
@@ -9124,6 +9587,9 @@ heap_redo(XLogReaderState *record)
case XLOG_HEAP_INSERT:
heap_xlog_insert(record);
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ heap_xlog_multi_insert(record);
+ break;
case XLOG_HEAP_DELETE:
heap_xlog_delete(record);
break;
@@ -9152,7 +9618,7 @@ heap2_redo(XLogReaderState *record)
{
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- switch (info & XLOG_HEAP_OPMASK)
+ switch (info & XLOG_HEAP2_OPMASK)
{
case XLOG_HEAP2_CLEAN:
heap_xlog_clean(record);
@@ -9166,9 +9632,6 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_VISIBLE:
heap_xlog_visible(record);
break;
- case XLOG_HEAP2_MULTI_INSERT:
- heap_xlog_multi_insert(record);
- break;
case XLOG_HEAP2_LOCK_UPDATED:
heap_xlog_lock_updated(record);
break;
@@ -9182,6 +9645,9 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_REWRITE:
heap_xlog_logical_rewrite(record);
break;
+ case XLOG_HEAP2_WARMCLEAR:
+ heap_xlog_warmclear(record);
+ break;
default:
elog(PANIC, "heap2_redo: unknown op code %u", info);
}
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index f54337c..4e8ed79 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -834,6 +834,13 @@ heap_get_root_tuples_internal(Page page, OffsetNumber target_offnum,
if (!HeapTupleHeaderIsHotUpdated(htup))
continue;
+ /*
+ * If the tuple has root line pointer, it must be the end of the
+ * chain
+ */
+ if (HeapTupleHeaderHasRootOffset(htup))
+ break;
+
/* Set up to scan the HOT-chain */
nextoffnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 2d3ae9b..bd469ee 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -404,6 +404,14 @@ rewrite_heap_tuple(RewriteState state,
old_tuple->t_data->t_infomask & HEAP_XACT_MASK;
/*
+ * We must clear the HEAP_WARM_TUPLE flag if the HEAP_WARM_UPDATED is
+ * cleared above.
+ */
+ if (HeapTupleHeaderIsWarmUpdated(old_tuple->t_data))
+ HeapTupleHeaderClearWarm(new_tuple->t_data);
+
+
+ /*
* While we have our hands on the tuple, we may as well freeze any
* eligible xmin or xmax, so that future VACUUM effort can be saved.
*/
@@ -428,7 +436,7 @@ rewrite_heap_tuple(RewriteState state,
memset(&hashkey, 0, sizeof(hashkey));
hashkey.xmin = HeapTupleHeaderGetUpdateXid(old_tuple->t_data);
- /*
+ /*
* We've already checked that this is not the last tuple in the chain,
* so fetch the next TID in the chain.
*/
@@ -737,7 +745,7 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
newitemid = PageGetItemId(page, newoff);
onpage_tup = (HeapTupleHeader) PageGetItem(page, newitemid);
- /*
+ /*
* Set t_ctid just to ensure that block number is copied correctly, but
* then immediately mark the tuple as the latest.
*/
diff --git a/src/backend/access/heap/tuptoaster.c b/src/backend/access/heap/tuptoaster.c
index 19e7048..47b01eb 100644
--- a/src/backend/access/heap/tuptoaster.c
+++ b/src/backend/access/heap/tuptoaster.c
@@ -1620,7 +1620,8 @@ toast_save_datum(Relation rel, Datum value,
toastrel,
toastidxs[i]->rd_index->indisunique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- NULL);
+ NULL,
+ false);
}
/*
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index a91fda7..d523c8f 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -127,6 +127,8 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys)
scan->xs_cbuf = InvalidBuffer;
scan->xs_continue_hot = false;
+ scan->indexInfo = NULL;
+
return scan;
}
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index cc5ac8b..04018fe 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -197,7 +197,8 @@ index_insert(Relation indexRelation,
ItemPointer heap_t_ctid,
Relation heapRelation,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo,
+ bool warm_update)
{
RELATION_CHECKS;
CHECK_REL_PROCEDURE(aminsert);
@@ -207,6 +208,12 @@ index_insert(Relation indexRelation,
(HeapTuple) NULL,
InvalidBuffer);
+ if (warm_update)
+ {
+ Assert(indexRelation->rd_amroutine->amwarminsert != NULL);
+ return indexRelation->rd_amroutine->amwarminsert(indexRelation, values,
+ isnull, heap_t_ctid, heapRelation, checkUnique, indexInfo);
+ }
return indexRelation->rd_amroutine->aminsert(indexRelation, values, isnull,
heap_t_ctid, heapRelation,
checkUnique, indexInfo);
@@ -234,6 +241,25 @@ index_beginscan(Relation heapRelation,
scan->heapRelation = heapRelation;
scan->xs_snapshot = snapshot;
+ /*
+ * If the index supports recheck, make sure that index tuple is saved
+ * during index scans. Also build and cache IndexInfo which is used by
+ * amrecheck routine.
+ *
+ * XXX Ideally, we should look at all indexes on the table and check if
+ * WARM is at all supported on the base table. If WARM is not supported
+ * then we don't need to do any recheck. RelationGetIndexAttrBitmap() does
+ * do that and sets rd_supportswarm after looking at all indexes. But we
+ * don't know if the function was called earlier in the session when we're
+ * here. We can't call it now because there exists a risk of causing
+ * deadlock.
+ */
+ if (indexRelation->rd_amroutine->amrecheck)
+ {
+ scan->xs_want_itup = true;
+ scan->indexInfo = BuildIndexInfo(indexRelation);
+ }
+
return scan;
}
@@ -358,6 +384,10 @@ index_endscan(IndexScanDesc scan)
if (scan->xs_temp_snap)
UnregisterSnapshot(scan->xs_snapshot);
+ /* Free cached IndexInfo, if any */
+ if (scan->indexInfo)
+ pfree(scan->indexInfo);
+
/* Release the scan data structure itself */
IndexScanEnd(scan);
}
@@ -535,7 +565,7 @@ index_getnext_tid(IndexScanDesc scan, ScanDirection direction)
/*
* The AM's amgettuple proc finds the next index entry matching the scan
* keys, and puts the TID into scan->xs_ctup.t_self. It should also set
- * scan->xs_recheck and possibly scan->xs_itup/scan->xs_hitup, though we
+ * scan->xs_tuple_recheck and possibly scan->xs_itup/scan->xs_hitup, though we
* pay no attention to those fields here.
*/
found = scan->indexRelation->rd_amroutine->amgettuple(scan, direction);
@@ -574,7 +604,7 @@ index_getnext_tid(IndexScanDesc scan, ScanDirection direction)
* dropped in a future index_getnext_tid, index_fetch_heap or index_endscan
* call).
*
- * Note: caller must check scan->xs_recheck, and perform rechecking of the
+ * Note: caller must check scan->xs_tuple_recheck, and perform rechecking of the
* scan keys if required. We do not do that here because we don't have
* enough information to do it efficiently in the general case.
* ----------------
@@ -585,6 +615,7 @@ index_fetch_heap(IndexScanDesc scan)
ItemPointer tid = &scan->xs_ctup.t_self;
bool all_dead = false;
bool got_heap_tuple;
+ bool tuple_recheck;
/* We can skip the buffer-switching logic if we're in mid-HOT chain. */
if (!scan->xs_continue_hot)
@@ -603,6 +634,8 @@ index_fetch_heap(IndexScanDesc scan)
heap_page_prune_opt(scan->heapRelation, scan->xs_cbuf);
}
+ tuple_recheck = false;
+
/* Obtain share-lock on the buffer so we can examine visibility */
LockBuffer(scan->xs_cbuf, BUFFER_LOCK_SHARE);
got_heap_tuple = heap_hot_search_buffer(tid, scan->heapRelation,
@@ -610,32 +643,60 @@ index_fetch_heap(IndexScanDesc scan)
scan->xs_snapshot,
&scan->xs_ctup,
&all_dead,
- !scan->xs_continue_hot);
- LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
+ !scan->xs_continue_hot,
+ &tuple_recheck);
if (got_heap_tuple)
{
+ bool res = true;
+
+ /*
+ * Ok we got a tuple which satisfies the snapshot, but if its part of a
+ * WARM chain, we must do additional checks to ensure that we are
+ * indeed returning a correct tuple. Note that if the index AM does not
+ * implement amrecheck method, then we don't any additional checks
+ * since WARM must have been disabled on such tables.
+ */
+ if (tuple_recheck && scan->xs_itup &&
+ scan->indexRelation->rd_amroutine->amrecheck)
+ {
+ res = scan->indexRelation->rd_amroutine->amrecheck(
+ scan->indexRelation,
+ scan->indexInfo,
+ scan->xs_itup,
+ scan->heapRelation,
+ &scan->xs_ctup);
+ }
+
+ LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
+
/*
* Only in a non-MVCC snapshot can more than one member of the HOT
* chain be visible.
*/
scan->xs_continue_hot = !IsMVCCSnapshot(scan->xs_snapshot);
pgstat_count_heap_fetch(scan->indexRelation);
- return &scan->xs_ctup;
+
+ if (res)
+ return &scan->xs_ctup;
}
+ else
+ {
+ LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
- /* We've reached the end of the HOT chain. */
- scan->xs_continue_hot = false;
+ /* We've reached the end of the HOT chain. */
+ scan->xs_continue_hot = false;
- /*
- * If we scanned a whole HOT chain and found only dead tuples, tell index
- * AM to kill its entry for that TID (this will take effect in the next
- * amgettuple call, in index_getnext_tid). We do not do this when in
- * recovery because it may violate MVCC to do so. See comments in
- * RelationGetIndexScan().
- */
- if (!scan->xactStartedInRecovery)
- scan->kill_prior_tuple = all_dead;
+ /*
+ * If we scanned a whole HOT chain and found only dead tuples, tell index
+ * AM to kill its entry for that TID (this will take effect in the next
+ * amgettuple call, in index_getnext_tid). We do not do this when in
+ * recovery because it may violate MVCC to do so. See comments in
+ * RelationGetIndexScan().
+ */
+ if (!scan->xactStartedInRecovery)
+ scan->kill_prior_tuple = all_dead;
+ }
return NULL;
}
diff --git a/src/backend/access/nbtree/nbtinsert.c b/src/backend/access/nbtree/nbtinsert.c
index 6dca810..328184b 100644
--- a/src/backend/access/nbtree/nbtinsert.c
+++ b/src/backend/access/nbtree/nbtinsert.c
@@ -20,12 +20,12 @@
#include "access/nbtxlog.h"
#include "access/transam.h"
#include "access/xloginsert.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
#include "utils/tqual.h"
-
typedef struct
{
/* context data for _bt_checksplitloc */
@@ -250,6 +250,10 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
BTPageOpaque opaque;
Buffer nbuf = InvalidBuffer;
bool found = false;
+ Buffer buffer;
+ HeapTupleData heapTuple;
+ bool recheck = false;
+ IndexInfo *indexInfo = BuildIndexInfo(rel);
/* Assume unique until we find a duplicate */
*is_unique = true;
@@ -309,6 +313,8 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
curitup = (IndexTuple) PageGetItem(page, curitemid);
htid = curitup->t_tid;
+ recheck = false;
+
/*
* If we are doing a recheck, we expect to find the tuple we
* are rechecking. It's not a duplicate, but we have to keep
@@ -326,112 +332,153 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
* have just a single index entry for the entire chain.
*/
else if (heap_hot_search(&htid, heapRel, &SnapshotDirty,
- &all_dead))
+ &all_dead, &recheck, &buffer,
+ &heapTuple))
{
TransactionId xwait;
+ bool result = true;
/*
- * It is a duplicate. If we are only doing a partial
- * check, then don't bother checking if the tuple is being
- * updated in another transaction. Just return the fact
- * that it is a potential conflict and leave the full
- * check till later.
+ * If the tuple was WARM update, we may again see our own
+ * tuple. Since WARM updates don't create new index
+ * entries, our own tuple is only reachable via the old
+ * index pointer.
*/
- if (checkUnique == UNIQUE_CHECK_PARTIAL)
+ if (checkUnique == UNIQUE_CHECK_EXISTING &&
+ ItemPointerCompare(&htid, &itup->t_tid) == 0)
{
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- *is_unique = false;
- return InvalidTransactionId;
+ found = true;
+ result = false;
+ if (recheck)
+ UnlockReleaseBuffer(buffer);
}
-
- /*
- * If this tuple is being updated by other transaction
- * then we have to wait for its commit/abort.
- */
- xwait = (TransactionIdIsValid(SnapshotDirty.xmin)) ?
- SnapshotDirty.xmin : SnapshotDirty.xmax;
-
- if (TransactionIdIsValid(xwait))
+ else if (recheck)
{
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- /* Tell _bt_doinsert to wait... */
- *speculativeToken = SnapshotDirty.speculativeToken;
- return xwait;
+ result = btrecheck(rel, indexInfo, curitup, heapRel, &heapTuple);
+ UnlockReleaseBuffer(buffer);
}
- /*
- * Otherwise we have a definite conflict. But before
- * complaining, look to see if the tuple we want to insert
- * is itself now committed dead --- if so, don't complain.
- * This is a waste of time in normal scenarios but we must
- * do it to support CREATE INDEX CONCURRENTLY.
- *
- * We must follow HOT-chains here because during
- * concurrent index build, we insert the root TID though
- * the actual tuple may be somewhere in the HOT-chain.
- * While following the chain we might not stop at the
- * exact tuple which triggered the insert, but that's OK
- * because if we find a live tuple anywhere in this chain,
- * we have a unique key conflict. The other live tuple is
- * not part of this chain because it had a different index
- * entry.
- */
- htid = itup->t_tid;
- if (heap_hot_search(&htid, heapRel, SnapshotSelf, NULL))
- {
- /* Normal case --- it's still live */
- }
- else
+ if (result)
{
/*
- * It's been deleted, so no error, and no need to
- * continue searching
+ * It is a duplicate. If we are only doing a partial
+ * check, then don't bother checking if the tuple is being
+ * updated in another transaction. Just return the fact
+ * that it is a potential conflict and leave the full
+ * check till later.
*/
- break;
- }
+ if (checkUnique == UNIQUE_CHECK_PARTIAL)
+ {
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ *is_unique = false;
+ return InvalidTransactionId;
+ }
- /*
- * Check for a conflict-in as we would if we were going to
- * write to this page. We aren't actually going to write,
- * but we want a chance to report SSI conflicts that would
- * otherwise be masked by this unique constraint
- * violation.
- */
- CheckForSerializableConflictIn(rel, NULL, buf);
+ /*
+ * If this tuple is being updated by other transaction
+ * then we have to wait for its commit/abort.
+ */
+ xwait = (TransactionIdIsValid(SnapshotDirty.xmin)) ?
+ SnapshotDirty.xmin : SnapshotDirty.xmax;
+
+ if (TransactionIdIsValid(xwait))
+ {
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ /* Tell _bt_doinsert to wait... */
+ *speculativeToken = SnapshotDirty.speculativeToken;
+ return xwait;
+ }
- /*
- * This is a definite conflict. Break the tuple down into
- * datums and report the error. But first, make sure we
- * release the buffer locks we're holding ---
- * BuildIndexValueDescription could make catalog accesses,
- * which in the worst case might touch this same index and
- * cause deadlocks.
- */
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- _bt_relbuf(rel, buf);
+ /*
+ * Otherwise we have a definite conflict. But before
+ * complaining, look to see if the tuple we want to insert
+ * is itself now committed dead --- if so, don't complain.
+ * This is a waste of time in normal scenarios but we must
+ * do it to support CREATE INDEX CONCURRENTLY.
+ *
+ * We must follow HOT-chains here because during
+ * concurrent index build, we insert the root TID though
+ * the actual tuple may be somewhere in the HOT-chain.
+ * While following the chain we might not stop at the
+ * exact tuple which triggered the insert, but that's OK
+ * because if we find a live tuple anywhere in this chain,
+ * we have a unique key conflict. The other live tuple is
+ * not part of this chain because it had a different index
+ * entry.
+ */
+ recheck = false;
+ ItemPointerCopy(&itup->t_tid, &htid);
+ if (heap_hot_search(&htid, heapRel, SnapshotSelf, NULL,
+ &recheck, &buffer, &heapTuple))
+ {
+ bool result = true;
+ if (recheck)
+ {
+ /*
+ * Recheck if the tuple actually satisfies the
+ * index key. Otherwise, we might be following
+ * a wrong index pointer and mustn't entertain
+ * this tuple.
+ */
+ result = btrecheck(rel, indexInfo, itup, heapRel, &heapTuple);
+ UnlockReleaseBuffer(buffer);
+ }
+ if (!result)
+ break;
+ /* Normal case --- it's still live */
+ }
+ else
+ {
+ /*
+ * It's been deleted, so no error, and no need to
+ * continue searching.
+ */
+ break;
+ }
- {
- Datum values[INDEX_MAX_KEYS];
- bool isnull[INDEX_MAX_KEYS];
- char *key_desc;
-
- index_deform_tuple(itup, RelationGetDescr(rel),
- values, isnull);
-
- key_desc = BuildIndexValueDescription(rel, values,
- isnull);
-
- ereport(ERROR,
- (errcode(ERRCODE_UNIQUE_VIOLATION),
- errmsg("duplicate key value violates unique constraint \"%s\"",
- RelationGetRelationName(rel)),
- key_desc ? errdetail("Key %s already exists.",
- key_desc) : 0,
- errtableconstraint(heapRel,
- RelationGetRelationName(rel))));
+ /*
+ * Check for a conflict-in as we would if we were going to
+ * write to this page. We aren't actually going to write,
+ * but we want a chance to report SSI conflicts that would
+ * otherwise be masked by this unique constraint
+ * violation.
+ */
+ CheckForSerializableConflictIn(rel, NULL, buf);
+
+ /*
+ * This is a definite conflict. Break the tuple down into
+ * datums and report the error. But first, make sure we
+ * release the buffer locks we're holding ---
+ * BuildIndexValueDescription could make catalog accesses,
+ * which in the worst case might touch this same index and
+ * cause deadlocks.
+ */
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ _bt_relbuf(rel, buf);
+
+ {
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ char *key_desc;
+
+ index_deform_tuple(itup, RelationGetDescr(rel),
+ values, isnull);
+
+ key_desc = BuildIndexValueDescription(rel, values,
+ isnull);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_UNIQUE_VIOLATION),
+ errmsg("duplicate key value violates unique constraint \"%s\"",
+ RelationGetRelationName(rel)),
+ key_desc ? errdetail("Key %s already exists.",
+ key_desc) : 0,
+ errtableconstraint(heapRel,
+ RelationGetRelationName(rel))));
+ }
}
}
else if (all_dead)
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index f815fd4..ce1bea0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -766,29 +766,20 @@ _bt_page_recyclable(Page page)
}
/*
- * Delete item(s) from a btree page during VACUUM.
+ * Delete item(s) and clear WARM item(s) on a btree page during VACUUM.
*
* This must only be used for deleting leaf items. Deleting an item on a
* non-leaf page has to be done as part of an atomic action that includes
- * deleting the page it points to.
+ * deleting the page it points to. We don't ever clear pointers on a non-leaf
+ * page.
*
* This routine assumes that the caller has pinned and locked the buffer.
* Also, the given itemnos *must* appear in increasing order in the array.
- *
- * We record VACUUMs and b-tree deletes differently in WAL. InHotStandby
- * we need to be able to pin all of the blocks in the btree in physical
- * order when replaying the effects of a VACUUM, just as we do for the
- * original VACUUM itself. lastBlockVacuumed allows us to tell whether an
- * intermediate range of blocks has had no changes at all by VACUUM,
- * and so must be scanned anyway during replay. We always write a WAL record
- * for the last block in the index, whether or not it contained any items
- * to be removed. This allows us to scan right up to end of index to
- * ensure correct locking.
*/
void
-_bt_delitems_vacuum(Relation rel, Buffer buf,
- OffsetNumber *itemnos, int nitems,
- BlockNumber lastBlockVacuumed)
+_bt_handleitems_vacuum(Relation rel, Buffer buf,
+ OffsetNumber *delitemnos, int ndelitems,
+ OffsetNumber *clearitemnos, int nclearitems)
{
Page page = BufferGetPage(buf);
BTPageOpaque opaque;
@@ -796,9 +787,20 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
/* No ereport(ERROR) until changes are logged */
START_CRIT_SECTION();
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the array and
+ * hence offnums recorded earlier won't make any sense after
+ * PageIndexMultiDelete is called.
+ */
+ if (nclearitems > 0)
+ _bt_clear_items(page, clearitemnos, nclearitems);
+
/* Fix the page */
- if (nitems > 0)
- PageIndexMultiDelete(page, itemnos, nitems);
+ if (ndelitems > 0)
+ PageIndexMultiDelete(page, delitemnos, ndelitems);
/*
* We can clear the vacuum cycle ID since this page has certainly been
@@ -824,7 +826,8 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
XLogRecPtr recptr;
xl_btree_vacuum xlrec_vacuum;
- xlrec_vacuum.lastBlockVacuumed = lastBlockVacuumed;
+ xlrec_vacuum.ndelitems = ndelitems;
+ xlrec_vacuum.nclearitems = nclearitems;
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);
@@ -835,8 +838,11 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
* is. When XLogInsert stores the whole buffer, the offsets array
* need not be stored too.
*/
- if (nitems > 0)
- XLogRegisterBufData(0, (char *) itemnos, nitems * sizeof(OffsetNumber));
+ if (ndelitems > 0)
+ XLogRegisterBufData(0, (char *) delitemnos, ndelitems * sizeof(OffsetNumber));
+
+ if (nclearitems > 0)
+ XLogRegisterBufData(0, (char *) clearitemnos, nclearitems * sizeof(OffsetNumber));
recptr = XLogInsert(RM_BTREE_ID, XLOG_BTREE_VACUUM);
@@ -1882,3 +1888,18 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, bool *rightsib_empty)
return true;
}
+
+void
+_bt_clear_items(Page page, OffsetNumber *clearitemnos, uint16 nclearitems)
+{
+ int i;
+ ItemId itemid;
+ IndexTuple itup;
+
+ for (i = 0; i < nclearitems; i++)
+ {
+ itemid = PageGetItemId(page, clearitemnos[i]);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ ItemPointerClearFlags(&itup->t_tid);
+ }
+}
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 775f2ff..6d558af 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -146,6 +146,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amwarminsert = btwarminsert;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -163,6 +164,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = btestimateparallelscan;
amroutine->aminitparallelscan = btinitparallelscan;
amroutine->amparallelrescan = btparallelrescan;
+ amroutine->amrecheck = btrecheck;
PG_RETURN_POINTER(amroutine);
}
@@ -315,11 +317,12 @@ btbuildempty(Relation index)
* Descend the tree recursively, find the appropriate location for our
* new tuple, and put it there.
*/
-bool
-btinsert(Relation rel, Datum *values, bool *isnull,
+static bool
+btinsert_internal(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo,
+ bool warm_update)
{
bool result;
IndexTuple itup;
@@ -328,6 +331,11 @@ btinsert(Relation rel, Datum *values, bool *isnull,
itup = index_form_tuple(RelationGetDescr(rel), values, isnull);
itup->t_tid = *ht_ctid;
+ if (warm_update)
+ ItemPointerSetFlags(&itup->t_tid, BTREE_INDEX_WARM_POINTER);
+ else
+ ItemPointerClearFlags(&itup->t_tid);
+
result = _bt_doinsert(rel, itup, checkUnique, heapRel);
pfree(itup);
@@ -335,6 +343,26 @@ btinsert(Relation rel, Datum *values, bool *isnull,
return result;
}
+bool
+btinsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return btinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, false);
+}
+
+bool
+btwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return btinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, true);
+}
+
/*
* btgettuple() -- Get the next tuple in the scan.
*/
@@ -1103,7 +1131,7 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
RBM_NORMAL, info->strategy);
LockBufferForCleanup(buf);
_bt_checkpage(rel, buf);
- _bt_delitems_vacuum(rel, buf, NULL, 0, vstate.lastBlockVacuumed);
+ _bt_handleitems_vacuum(rel, buf, NULL, 0, NULL, 0);
_bt_relbuf(rel, buf);
}
@@ -1201,6 +1229,8 @@ restart:
{
OffsetNumber deletable[MaxOffsetNumber];
int ndeletable;
+ OffsetNumber clearwarm[MaxOffsetNumber];
+ int nclearwarm;
OffsetNumber offnum,
minoff,
maxoff;
@@ -1239,7 +1269,7 @@ restart:
* Scan over all items to see which ones need deleted according to the
* callback function.
*/
- ndeletable = 0;
+ ndeletable = nclearwarm = 0;
minoff = P_FIRSTDATAKEY(opaque);
maxoff = PageGetMaxOffsetNumber(page);
if (callback)
@@ -1250,6 +1280,9 @@ restart:
{
IndexTuple itup;
ItemPointer htup;
+ int flags;
+ bool is_warm = false;
+ IndexBulkDeleteCallbackResult result;
itup = (IndexTuple) PageGetItem(page,
PageGetItemId(page, offnum));
@@ -1276,16 +1309,36 @@ restart:
* applies to *any* type of index that marks index tuples as
* killed.
*/
- if (callback(htup, callback_state))
+ flags = ItemPointerGetFlags(&itup->t_tid);
+ is_warm = ((flags & BTREE_INDEX_WARM_POINTER) != 0);
+
+ if (is_warm)
+ stats->num_warm_pointers++;
+ else
+ stats->num_clear_pointers++;
+
+ result = callback(htup, is_warm, callback_state);
+ if (result == IBDCR_DELETE)
+ {
+ if (is_warm)
+ stats->warm_pointers_removed++;
+ else
+ stats->clear_pointers_removed++;
deletable[ndeletable++] = offnum;
+ }
+ else if (result == IBDCR_CLEAR_WARM)
+ {
+ clearwarm[nclearwarm++] = offnum;
+ }
}
}
/*
- * Apply any needed deletes. We issue just one _bt_delitems_vacuum()
- * call per page, so as to minimize WAL traffic.
+ * Apply any needed deletes and clearing. We issue just one
+ * _bt_handleitems_vacuum() call per page, so as to minimize WAL
+ * traffic.
*/
- if (ndeletable > 0)
+ if (ndeletable > 0 || nclearwarm > 0)
{
/*
* Notice that the issued XLOG_BTREE_VACUUM WAL record includes
@@ -1301,8 +1354,8 @@ restart:
* doesn't seem worth the amount of bookkeeping it'd take to avoid
* that.
*/
- _bt_delitems_vacuum(rel, buf, deletable, ndeletable,
- vstate->lastBlockVacuumed);
+ _bt_handleitems_vacuum(rel, buf, deletable, ndeletable,
+ clearwarm, nclearwarm);
/*
* Remember highest leaf page number we've issued a
@@ -1312,6 +1365,7 @@ restart:
vstate->lastBlockVacuumed = blkno;
stats->tuples_removed += ndeletable;
+ stats->pointers_cleared += nclearwarm;
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index 5b259a3..8dab5a8 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -20,11 +20,13 @@
#include "access/nbtree.h"
#include "access/reloptions.h"
#include "access/relscan.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/datum.h"
typedef struct BTSortArrayContext
@@ -2069,3 +2071,64 @@ btproperty(Oid index_oid, int attno,
return false; /* punt to generic code */
}
}
+
+/*
+ * Check if the index tuple's key matches the one computed from the given heap
+ * tuple's attribute
+ */
+bool
+btrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
+{
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ bool isavail[INDEX_MAX_KEYS];
+ int i;
+ bool equal;
+ int natts = indexRel->rd_rel->relnatts;
+ Form_pg_attribute att;
+
+ FormIndexPlainDatum(indexInfo, heapRel, heapTuple, values, isnull, isavail);
+
+ equal = true;
+ for (i = 1; i <= natts; i++)
+ {
+ Datum indxvalue;
+ bool indxisnull;
+
+ /* No need to compare if the attribute value is not available */
+ if (!isavail[i - 1])
+ continue;
+
+ indxvalue = index_getattr(indexTuple, i, indexRel->rd_att, &indxisnull);
+
+ /*
+ * If both are NULL, then they are equal
+ */
+ if (isnull[i - 1] && indxisnull)
+ continue;
+
+ /*
+ * If just one is NULL, then they are not equal
+ */
+ if (isnull[i - 1] || indxisnull)
+ {
+ equal = false;
+ break;
+ }
+
+ /*
+ * Now just do a raw memory comparison. If the index tuple was formed
+ * using this heap tuple, the computed index values must match
+ */
+ att = indexRel->rd_att->attrs[i - 1];
+ if (!datumIsEqual(values[i - 1], indxvalue, att->attbyval,
+ att->attlen))
+ {
+ equal = false;
+ break;
+ }
+ }
+
+ return equal;
+}
diff --git a/src/backend/access/nbtree/nbtxlog.c b/src/backend/access/nbtree/nbtxlog.c
index ac60db0..92be5c8 100644
--- a/src/backend/access/nbtree/nbtxlog.c
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -390,83 +390,9 @@ btree_xlog_vacuum(XLogReaderState *record)
Buffer buffer;
Page page;
BTPageOpaque opaque;
-#ifdef UNUSED
xl_btree_vacuum *xlrec = (xl_btree_vacuum *) XLogRecGetData(record);
/*
- * This section of code is thought to be no longer needed, after analysis
- * of the calling paths. It is retained to allow the code to be reinstated
- * if a flaw is revealed in that thinking.
- *
- * If we are running non-MVCC scans using this index we need to do some
- * additional work to ensure correctness, which is known as a "pin scan"
- * described in more detail in next paragraphs. We used to do the extra
- * work in all cases, whereas we now avoid that work in most cases. If
- * lastBlockVacuumed is set to InvalidBlockNumber then we skip the
- * additional work required for the pin scan.
- *
- * Avoiding this extra work is important since it requires us to touch
- * every page in the index, so is an O(N) operation. Worse, it is an
- * operation performed in the foreground during redo, so it delays
- * replication directly.
- *
- * If queries might be active then we need to ensure every leaf page is
- * unpinned between the lastBlockVacuumed and the current block, if there
- * are any. This prevents replay of the VACUUM from reaching the stage of
- * removing heap tuples while there could still be indexscans "in flight"
- * to those particular tuples for those scans which could be confused by
- * finding new tuples at the old TID locations (see nbtree/README).
- *
- * It might be worth checking if there are actually any backends running;
- * if not, we could just skip this.
- *
- * Since VACUUM can visit leaf pages out-of-order, it might issue records
- * with lastBlockVacuumed >= block; that's not an error, it just means
- * nothing to do now.
- *
- * Note: since we touch all pages in the range, we will lock non-leaf
- * pages, and also any empty (all-zero) pages that may be in the index. It
- * doesn't seem worth the complexity to avoid that. But it's important
- * that HotStandbyActiveInReplay() will not return true if the database
- * isn't yet consistent; so we need not fear reading still-corrupt blocks
- * here during crash recovery.
- */
- if (HotStandbyActiveInReplay() && BlockNumberIsValid(xlrec->lastBlockVacuumed))
- {
- RelFileNode thisrnode;
- BlockNumber thisblkno;
- BlockNumber blkno;
-
- XLogRecGetBlockTag(record, 0, &thisrnode, NULL, &thisblkno);
-
- for (blkno = xlrec->lastBlockVacuumed + 1; blkno < thisblkno; blkno++)
- {
- /*
- * We use RBM_NORMAL_NO_LOG mode because it's not an error
- * condition to see all-zero pages. The original btvacuumpage
- * scan would have skipped over all-zero pages, noting them in FSM
- * but not bothering to initialize them just yet; so we mustn't
- * throw an error here. (We could skip acquiring the cleanup lock
- * if PageIsNew, but it's probably not worth the cycles to test.)
- *
- * XXX we don't actually need to read the block, we just need to
- * confirm it is unpinned. If we had a special call into the
- * buffer manager we could optimise this so that if the block is
- * not in shared_buffers we confirm it as unpinned. Optimizing
- * this is now moot, since in most cases we avoid the scan.
- */
- buffer = XLogReadBufferExtended(thisrnode, MAIN_FORKNUM, blkno,
- RBM_NORMAL_NO_LOG);
- if (BufferIsValid(buffer))
- {
- LockBufferForCleanup(buffer);
- UnlockReleaseBuffer(buffer);
- }
- }
- }
-#endif
-
- /*
* Like in btvacuumpage(), we need to take a cleanup lock on every leaf
* page. See nbtree/README for details.
*/
@@ -482,19 +408,30 @@ btree_xlog_vacuum(XLogReaderState *record)
if (len > 0)
{
- OffsetNumber *unused;
- OffsetNumber *unend;
+ OffsetNumber *offnums = (OffsetNumber *) ptr;
- unused = (OffsetNumber *) ptr;
- unend = (OffsetNumber *) ((char *) ptr + len);
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the
+ * array and hence offnums recorded earlier won't make any sense
+ * after PageIndexMultiDelete is called.
+ */
+ if (xlrec->nclearitems > 0)
+ _bt_clear_items(page, offnums + xlrec->ndelitems,
+ xlrec->nclearitems);
- if ((unend - unused) > 0)
- PageIndexMultiDelete(page, unused, unend - unused);
+ /*
+ * And handle the deleted items too
+ */
+ if (xlrec->ndelitems > 0)
+ PageIndexMultiDelete(page, offnums, xlrec->ndelitems);
}
/*
* Mark the page as not containing any LP_DEAD items --- see comments
- * in _bt_delitems_vacuum().
+ * in _bt_handleitems_vacuum().
*/
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
opaque->btpo_flags &= ~BTP_HAS_GARBAGE;
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 44d2d63..d373e61 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -44,6 +44,12 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
+ else if (info == XLOG_HEAP_MULTI_INSERT)
+ {
+ xl_heap_multi_insert *xlrec = (xl_heap_multi_insert *) rec;
+
+ appendStringInfo(buf, "%d tuples", xlrec->ntuples);
+ }
else if (info == XLOG_HEAP_DELETE)
{
xl_heap_delete *xlrec = (xl_heap_delete *) rec;
@@ -102,7 +108,7 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- info &= XLOG_HEAP_OPMASK;
+ info &= XLOG_HEAP2_OPMASK;
if (info == XLOG_HEAP2_CLEAN)
{
xl_heap_clean *xlrec = (xl_heap_clean *) rec;
@@ -129,12 +135,6 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "cutoff xid %u flags %d",
xlrec->cutoff_xid, xlrec->flags);
}
- else if (info == XLOG_HEAP2_MULTI_INSERT)
- {
- xl_heap_multi_insert *xlrec = (xl_heap_multi_insert *) rec;
-
- appendStringInfo(buf, "%d tuples", xlrec->ntuples);
- }
else if (info == XLOG_HEAP2_LOCK_UPDATED)
{
xl_heap_lock_updated *xlrec = (xl_heap_lock_updated *) rec;
@@ -171,6 +171,12 @@ heap_identify(uint8 info)
case XLOG_HEAP_INSERT | XLOG_HEAP_INIT_PAGE:
id = "INSERT+INIT";
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ id = "MULTI_INSERT";
+ break;
+ case XLOG_HEAP_MULTI_INSERT | XLOG_HEAP_INIT_PAGE:
+ id = "MULTI_INSERT+INIT";
+ break;
case XLOG_HEAP_DELETE:
id = "DELETE";
break;
@@ -219,12 +225,6 @@ heap2_identify(uint8 info)
case XLOG_HEAP2_VISIBLE:
id = "VISIBLE";
break;
- case XLOG_HEAP2_MULTI_INSERT:
- id = "MULTI_INSERT";
- break;
- case XLOG_HEAP2_MULTI_INSERT | XLOG_HEAP_INIT_PAGE:
- id = "MULTI_INSERT+INIT";
- break;
case XLOG_HEAP2_LOCK_UPDATED:
id = "LOCK_UPDATED";
break;
diff --git a/src/backend/access/rmgrdesc/nbtdesc.c b/src/backend/access/rmgrdesc/nbtdesc.c
index fbde9d6..6b2c5d6 100644
--- a/src/backend/access/rmgrdesc/nbtdesc.c
+++ b/src/backend/access/rmgrdesc/nbtdesc.c
@@ -48,8 +48,8 @@ btree_desc(StringInfo buf, XLogReaderState *record)
{
xl_btree_vacuum *xlrec = (xl_btree_vacuum *) rec;
- appendStringInfo(buf, "lastBlockVacuumed %u",
- xlrec->lastBlockVacuumed);
+ appendStringInfo(buf, "ndelitems %u, nclearitems %u",
+ xlrec->ndelitems, xlrec->nclearitems);
break;
}
case XLOG_BTREE_DELETE:
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index e57ac49..59ef7f3 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -72,6 +72,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index cce9b3f..711d351 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -155,7 +155,8 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
{
Assert(ItemPointerIsValid(<->heapPtr));
- if (bds->callback(<->heapPtr, bds->callback_state))
+ if (bds->callback(<->heapPtr, false, bds->callback_state) ==
+ IBDCR_DELETE)
{
bds->stats->tuples_removed += 1;
deletable[i] = true;
@@ -425,7 +426,8 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
{
Assert(ItemPointerIsValid(<->heapPtr));
- if (bds->callback(<->heapPtr, bds->callback_state))
+ if (bds->callback(<->heapPtr, false, bds->callback_state) ==
+ IBDCR_DELETE)
{
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
@@ -902,10 +904,10 @@ spgbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
}
/* Dummy callback to delete no tuples during spgvacuumcleanup */
-static bool
-dummy_callback(ItemPointer itemptr, void *state)
+static IndexBulkDeleteCallbackResult
+dummy_callback(ItemPointer itemptr, bool is_warm, void *state)
{
- return false;
+ return IBDCR_KEEP;
}
/*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8d42a34..67e68d1 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -54,6 +54,7 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "optimizer/clauses.h"
+#include "optimizer/var.h"
#include "parser/parser.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
@@ -114,7 +115,7 @@ static void IndexCheckExclusion(Relation heapRelation,
IndexInfo *indexInfo);
static inline int64 itemptr_encode(ItemPointer itemptr);
static inline void itemptr_decode(ItemPointer itemptr, int64 encoded);
-static bool validate_index_callback(ItemPointer itemptr, void *opaque);
+static IndexBulkDeleteCallbackResult validate_index_callback(ItemPointer itemptr, bool is_warm, void *opaque);
static void validate_index_heapscan(Relation heapRelation,
Relation indexRelation,
IndexInfo *indexInfo,
@@ -1691,6 +1692,20 @@ BuildIndexInfo(Relation index)
ii->ii_AmCache = NULL;
ii->ii_Context = CurrentMemoryContext;
+ /* build a bitmap of all table attributes referred by this index */
+ for (i = 0; i < ii->ii_NumIndexAttrs; i++)
+ {
+ AttrNumber attr = ii->ii_KeyAttrNumbers[i];
+ ii->ii_indxattrs = bms_add_member(ii->ii_indxattrs, attr -
+ FirstLowInvalidHeapAttributeNumber);
+ }
+
+ /* Collect all attributes used in expressions, too */
+ pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_indxattrs);
+
+ /* Collect all attributes in the index predicate, too */
+ pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_indxattrs);
+
return ii;
}
@@ -1816,6 +1831,50 @@ FormIndexDatum(IndexInfo *indexInfo,
elog(ERROR, "wrong number of index expressions");
}
+/*
+ * This is same as FormIndexDatum but we don't compute any expression
+ * attributes and hence can be used when executor interfaces are not available.
+ * If i'th attribute is available then isavail[i] is set to true, else set to
+ * false. The caller must always check if an attribute value is available
+ * before trying to do anything useful with that.
+ */
+void
+FormIndexPlainDatum(IndexInfo *indexInfo,
+ Relation heapRel,
+ HeapTuple heapTup,
+ Datum *values,
+ bool *isnull,
+ bool *isavail)
+{
+ int i;
+
+ for (i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
+ {
+ int keycol = indexInfo->ii_KeyAttrNumbers[i];
+ Datum iDatum;
+ bool isNull;
+
+ if (keycol != 0)
+ {
+ /*
+ * Plain index column; get the value we need directly from the
+ * heap tuple.
+ */
+ iDatum = heap_getattr(heapTup, keycol, RelationGetDescr(heapRel), &isNull);
+ values[i] = iDatum;
+ isnull[i] = isNull;
+ isavail[i] = true;
+ }
+ else
+ {
+ /*
+ * This is an expression attribute which can't be computed by us.
+ * So just inform the caller about it.
+ */
+ isavail[i] = false;
+ }
+ }
+}
/*
* index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX
@@ -2934,15 +2993,15 @@ itemptr_decode(ItemPointer itemptr, int64 encoded)
/*
* validate_index_callback - bulkdelete callback to collect the index TIDs
*/
-static bool
-validate_index_callback(ItemPointer itemptr, void *opaque)
+static IndexBulkDeleteCallbackResult
+validate_index_callback(ItemPointer itemptr, bool is_warm, void *opaque)
{
v_i_state *state = (v_i_state *) opaque;
int64 encoded = itemptr_encode(itemptr);
tuplesort_putdatum(state->tuplesort, Int64GetDatum(encoded), false);
state->itups += 1;
- return false; /* never actually delete anything */
+ return IBDCR_KEEP; /* never actually delete anything */
}
/*
@@ -3163,7 +3222,8 @@ validate_index_heapscan(Relation heapRelation,
heapRelation,
indexInfo->ii_Unique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- indexInfo);
+ indexInfo,
+ false);
state->tups_inserted += 1;
}
diff --git a/src/backend/catalog/indexing.c b/src/backend/catalog/indexing.c
index abc344a..6392f33 100644
--- a/src/backend/catalog/indexing.c
+++ b/src/backend/catalog/indexing.c
@@ -66,10 +66,15 @@ CatalogCloseIndexes(CatalogIndexState indstate)
*
* This should be called for each inserted or updated catalog tuple.
*
+ * If the tuple was WARM updated, the modified_attrs contains the list of
+ * columns updated by the update. We must not insert new index entries for
+ * indexes which do not refer to any of the modified columns.
+ *
* This is effectively a cut-down version of ExecInsertIndexTuples.
*/
static void
-CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
+CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple,
+ Bitmapset *modified_attrs, bool warm_update)
{
int i;
int numIndexes;
@@ -79,12 +84,28 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
IndexInfo **indexInfoArray;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
+ ItemPointerData root_tid;
- /* HOT update does not require index inserts */
- if (HeapTupleIsHeapOnly(heapTuple))
+ /*
+ * HOT update does not require index inserts, but WARM may need for some
+ * indexes.
+ */
+ if (HeapTupleIsHeapOnly(heapTuple) && !warm_update)
return;
/*
+ * If we've done a WARM update, then we must index the TID of the root line
+ * pointer and not the actual TID of the new tuple.
+ */
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(heapTuple->t_self)),
+ HeapTupleHeaderGetRootOffset(heapTuple->t_data));
+ else
+ ItemPointerCopy(&heapTuple->t_self, &root_tid);
+
+
+ /*
* Get information from the state structure. Fall out if nothing to do.
*/
numIndexes = indstate->ri_NumIndices;
@@ -112,6 +133,17 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
continue;
/*
+ * If we've done WARM update, then we must not insert a new index tuple
+ * if none of the index keys have changed. This is not just an
+ * optimization, but a requirement for WARM to work correctly.
+ */
+ if (warm_update)
+ {
+ if (!bms_overlap(modified_attrs, indexInfo->ii_indxattrs))
+ continue;
+ }
+
+ /*
* Expressional and partial indexes on system catalogs are not
* supported, nor exclusion constraints, nor deferred uniqueness
*/
@@ -136,11 +168,12 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
index_insert(relationDescs[i], /* index relation */
values, /* array of index Datums */
isnull, /* is-null flags */
- &(heapTuple->t_self), /* tid of heap tuple */
+ &root_tid,
heapRelation,
relationDescs[i]->rd_index->indisunique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- indexInfo);
+ indexInfo,
+ warm_update);
}
ExecDropSingleTupleTableSlot(slot);
@@ -168,7 +201,7 @@ CatalogTupleInsert(Relation heapRel, HeapTuple tup)
oid = simple_heap_insert(heapRel, tup);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, NULL, false);
CatalogCloseIndexes(indstate);
return oid;
@@ -190,7 +223,7 @@ CatalogTupleInsertWithInfo(Relation heapRel, HeapTuple tup,
oid = simple_heap_insert(heapRel, tup);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, NULL, false);
return oid;
}
@@ -210,12 +243,14 @@ void
CatalogTupleUpdate(Relation heapRel, ItemPointer otid, HeapTuple tup)
{
CatalogIndexState indstate;
+ bool warm_update;
+ Bitmapset *modified_attrs;
indstate = CatalogOpenIndexes(heapRel);
- simple_heap_update(heapRel, otid, tup);
+ simple_heap_update(heapRel, otid, tup, &modified_attrs, &warm_update);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, modified_attrs, warm_update);
CatalogCloseIndexes(indstate);
}
@@ -231,9 +266,12 @@ void
CatalogTupleUpdateWithInfo(Relation heapRel, ItemPointer otid, HeapTuple tup,
CatalogIndexState indstate)
{
- simple_heap_update(heapRel, otid, tup);
+ Bitmapset *modified_attrs;
+ bool warm_update;
+
+ simple_heap_update(heapRel, otid, tup, &modified_attrs, &warm_update);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, modified_attrs, warm_update);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b6552da..15d0fe4 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -498,6 +498,7 @@ CREATE VIEW pg_stat_all_tables AS
pg_stat_get_tuples_updated(C.oid) AS n_tup_upd,
pg_stat_get_tuples_deleted(C.oid) AS n_tup_del,
pg_stat_get_tuples_hot_updated(C.oid) AS n_tup_hot_upd,
+ pg_stat_get_tuples_warm_updated(C.oid) AS n_tup_warm_upd,
pg_stat_get_live_tuples(C.oid) AS n_live_tup,
pg_stat_get_dead_tuples(C.oid) AS n_dead_tup,
pg_stat_get_mod_since_analyze(C.oid) AS n_mod_since_analyze,
@@ -528,7 +529,8 @@ CREATE VIEW pg_stat_xact_all_tables AS
pg_stat_get_xact_tuples_inserted(C.oid) AS n_tup_ins,
pg_stat_get_xact_tuples_updated(C.oid) AS n_tup_upd,
pg_stat_get_xact_tuples_deleted(C.oid) AS n_tup_del,
- pg_stat_get_xact_tuples_hot_updated(C.oid) AS n_tup_hot_upd
+ pg_stat_get_xact_tuples_hot_updated(C.oid) AS n_tup_hot_upd,
+ pg_stat_get_xact_tuples_warm_updated(C.oid) AS n_tup_warm_upd
FROM pg_class C LEFT JOIN
pg_index I ON C.oid = I.indrelid
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git a/src/backend/commands/constraint.c b/src/backend/commands/constraint.c
index e2544e5..330b661 100644
--- a/src/backend/commands/constraint.c
+++ b/src/backend/commands/constraint.c
@@ -40,6 +40,7 @@ unique_key_recheck(PG_FUNCTION_ARGS)
TriggerData *trigdata = castNode(TriggerData, fcinfo->context);
const char *funcname = "unique_key_recheck";
HeapTuple new_row;
+ HeapTupleData heapTuple;
ItemPointerData tmptid;
Relation indexRel;
IndexInfo *indexInfo;
@@ -102,7 +103,8 @@ unique_key_recheck(PG_FUNCTION_ARGS)
* removed.
*/
tmptid = new_row->t_self;
- if (!heap_hot_search(&tmptid, trigdata->tg_relation, SnapshotSelf, NULL))
+ if (!heap_hot_search(&tmptid, trigdata->tg_relation, SnapshotSelf, NULL,
+ NULL, NULL, &heapTuple))
{
/*
* All rows in the HOT chain are dead, so skip the check.
@@ -166,7 +168,8 @@ unique_key_recheck(PG_FUNCTION_ARGS)
*/
index_insert(indexRel, values, isnull, &(new_row->t_self),
trigdata->tg_relation, UNIQUE_CHECK_EXISTING,
- indexInfo);
+ indexInfo,
+ false);
}
else
{
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index ba89b29..120e261 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -2681,6 +2681,8 @@ CopyFrom(CopyState cstate)
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot,
&(tuple->t_self),
+ &(tuple->t_self),
+ NULL,
estate,
false,
NULL,
@@ -2835,6 +2837,7 @@ CopyFromInsertBatch(CopyState cstate, EState *estate, CommandId mycid,
ExecStoreTuple(bufferedTuples[i], myslot, InvalidBuffer, false);
recheckIndexes =
ExecInsertIndexTuples(myslot, &(bufferedTuples[i]->t_self),
+ &(bufferedTuples[i]->t_self), NULL,
estate, false, NULL, NIL);
ExecARInsertTriggers(estate, resultRelInfo,
bufferedTuples[i],
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 9618032..1b2abd4 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -694,7 +694,14 @@ DefineIndex(Oid relationId,
* visible to other transactions before we start to build the index. That
* will prevent them from making incompatible HOT updates. The new index
* will be marked not indisready and not indisvalid, so that no one else
- * tries to either insert into it or use it for queries.
+ * tries to either insert into it or use it for queries. In addition to
+ * that, WARM updates will be disallowed if an update is modifying one of
+ * the columns used by this new index. This is necessary to ensure that we
+ * don't create WARM tuples which do not have corresponding entry in this
+ * index. It must be noted that during the second phase, we will index only
+ * those heap tuples whose root line pointer is not already in the index,
+ * hence it's important that all tuples in a given chain, has the same
+ * value for any indexed column (including this new index).
*
* We must commit our current transaction so that the index becomes
* visible; then start another. Note that all the data structures we just
@@ -742,7 +749,10 @@ DefineIndex(Oid relationId,
* marked as "not-ready-for-inserts". The index is consulted while
* deciding HOT-safety though. This arrangement ensures that no new HOT
* chains can be created where the new tuple and the old tuple in the
- * chain have different index keys.
+ * chain have different index keys. Also, the new index is consulted for
+ * deciding whether a WARM update is possible, and WARM update is not done
+ * if a column used by this index is being updated. This ensures that we
+ * don't create WARM tuples which are not indexed by this index.
*
* We now take a new snapshot, and build the index using all tuples that
* are visible in this snapshot. We can be sure that any HOT updates to
@@ -777,7 +787,8 @@ DefineIndex(Oid relationId,
/*
* Update the pg_index row to mark the index as ready for inserts. Once we
* commit this transaction, any new transactions that open the table must
- * insert new entries into the index for insertions and non-HOT updates.
+ * insert new entries into the index for insertions and non-HOT updates or
+ * WARM updates where this index needs new entry.
*/
index_set_state_flags(indexRelationId, INDEX_CREATE_SET_READY);
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index b74e493..2b054f7 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -104,6 +104,39 @@
*/
#define PREFETCH_SIZE ((BlockNumber) 32)
+/*
+ * Structure to track WARM chains that can be converted into HOT chains during
+ * this run.
+ *
+ * To reduce space requirement, we're using bitfields. But the way things are
+ * laid down, we're still wasting 1-byte per candidate chain.
+ */
+typedef struct LVWarmChain
+{
+ ItemPointerData chain_tid; /* root of the chain */
+
+ /*
+ * 1 - if the chain contains only post-warm tuples
+ * 0 - if the chain contains only pre-warm tuples
+ */
+ uint8 is_postwarm_chain:2;
+
+ /* 1 - if this chain can't be cleared of WARM tuples */
+ uint8 keep_warm_chain:2;
+
+ /*
+ * Number of CLEAR pointers to this root TID found so far - must never be
+ * more than 2.
+ */
+ uint8 num_clear_pointers:2;
+
+ /*
+ * Number of WARM pointers to this root TID found so far - must never be
+ * more than 1.
+ */
+ uint8 num_warm_pointers:2;
+} LVWarmChain;
+
typedef struct LVRelStats
{
/* hasindex = true means two-pass strategy; false means one-pass */
@@ -122,6 +155,14 @@ typedef struct LVRelStats
BlockNumber pages_removed;
double tuples_deleted;
BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
+
+ /* List of candidate WARM chains that can be converted into HOT chains */
+ /* NB: this list is ordered by TID of the root pointers */
+ int num_warm_chains; /* current # of entries */
+ int max_warm_chains; /* # slots allocated in array */
+ LVWarmChain *warm_chains; /* array of LVWarmChain */
+ double num_non_convertible_warm_chains;
+
/* List of TIDs of tuples we intend to delete */
/* NB: this list is ordered by TID address */
int num_dead_tuples; /* current # of entries */
@@ -150,6 +191,7 @@ static void lazy_scan_heap(Relation onerel, int options,
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
static void lazy_vacuum_index(Relation indrel,
+ bool clear_warm,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
static void lazy_cleanup_index(Relation indrel,
@@ -157,6 +199,10 @@ static void lazy_cleanup_index(Relation indrel,
LVRelStats *vacrelstats);
static int lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
int tupindex, LVRelStats *vacrelstats, Buffer *vmbuffer);
+static int lazy_warmclear_page(Relation onerel, BlockNumber blkno,
+ Buffer buffer, int chainindex, LVRelStats *vacrelstats,
+ Buffer *vmbuffer, bool check_all_visible);
+static void lazy_reset_warm_pointer_count(LVRelStats *vacrelstats);
static bool should_attempt_truncation(LVRelStats *vacrelstats);
static void lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats);
static BlockNumber count_nondeletable_pages(Relation onerel,
@@ -164,8 +210,15 @@ static BlockNumber count_nondeletable_pages(Relation onerel,
static void lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks);
static void lazy_record_dead_tuple(LVRelStats *vacrelstats,
ItemPointer itemptr);
-static bool lazy_tid_reaped(ItemPointer itemptr, void *state);
+static void lazy_record_warm_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr);
+static void lazy_record_clear_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr);
+static IndexBulkDeleteCallbackResult lazy_tid_reaped(ItemPointer itemptr, bool is_warm, void *state);
+static IndexBulkDeleteCallbackResult lazy_indexvac_phase1(ItemPointer itemptr, bool is_warm, void *state);
+static IndexBulkDeleteCallbackResult lazy_indexvac_phase2(ItemPointer itemptr, bool is_warm, void *state);
static int vac_cmp_itemptr(const void *left, const void *right);
+static int vac_cmp_warm_chain(const void *left, const void *right);
static bool heap_page_is_all_visible(Relation rel, Buffer buf,
TransactionId *visibility_cutoff_xid, bool *all_frozen);
@@ -690,8 +743,10 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* If we are close to overrunning the available space for dead-tuple
* TIDs, pause and do a cycle of vacuuming before we tackle this page.
*/
- if ((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
- vacrelstats->num_dead_tuples > 0)
+ if (((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
+ vacrelstats->num_dead_tuples > 0) ||
+ ((vacrelstats->max_warm_chains - vacrelstats->num_warm_chains) < MaxHeapTuplesPerPage &&
+ vacrelstats->num_warm_chains > 0))
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
@@ -721,6 +776,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
+ (vacrelstats->num_warm_chains > 0),
&indstats[i],
vacrelstats);
@@ -743,6 +799,9 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* valid.
*/
vacrelstats->num_dead_tuples = 0;
+ vacrelstats->num_warm_chains = 0;
+ memset(vacrelstats->warm_chains, 0,
+ vacrelstats->max_warm_chains * sizeof (LVWarmChain));
vacrelstats->num_index_scans++;
/* Report that we are once again scanning the heap */
@@ -947,15 +1006,31 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
continue;
}
+ ItemPointerSet(&(tuple.t_self), blkno, offnum);
+
/* Redirect items mustn't be touched */
if (ItemIdIsRedirected(itemid))
{
+ HeapCheckWarmChainStatus status = heap_check_warm_chain(page,
+ &tuple.t_self, false);
+ if (HCWC_IS_WARM_UPDATED(status))
+ {
+ /*
+ * A chain which is either complete WARM or CLEAR is a
+ * candidate for chain conversion. Remember the chain and
+ * whether the chain has all WARM tuples or not.
+ */
+ if (HCWC_IS_ALL_WARM(status))
+ lazy_record_warm_chain(vacrelstats, &tuple.t_self);
+ else if (HCWC_IS_ALL_CLEAR(status))
+ lazy_record_clear_chain(vacrelstats, &tuple.t_self);
+ else
+ vacrelstats->num_non_convertible_warm_chains++;
+ }
hastup = true; /* this page won't be truncatable */
continue;
}
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
-
/*
* DEAD item pointers are to be vacuumed normally; but we don't
* count them in tups_vacuumed, else we'd be double-counting (at
@@ -975,6 +1050,26 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
tuple.t_len = ItemIdGetLength(itemid);
tuple.t_tableOid = RelationGetRelid(onerel);
+ if (!HeapTupleIsHeapOnly(&tuple))
+ {
+ HeapCheckWarmChainStatus status = heap_check_warm_chain(page,
+ &tuple.t_self, false);
+ if (HCWC_IS_WARM_UPDATED(status))
+ {
+ /*
+ * A chain which is either complete WARM or CLEAR is a
+ * candidate for chain conversion. Remember the chain and
+ * its color.
+ */
+ if (HCWC_IS_ALL_WARM(status))
+ lazy_record_warm_chain(vacrelstats, &tuple.t_self);
+ else if (HCWC_IS_ALL_CLEAR(status))
+ lazy_record_clear_chain(vacrelstats, &tuple.t_self);
+ else
+ vacrelstats->num_non_convertible_warm_chains++;
+ }
+ }
+
tupgone = false;
switch (HeapTupleSatisfiesVacuum(&tuple, OldestXmin, buf))
@@ -1040,6 +1135,19 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
break;
}
+ /*
+ * If this tuple was ever WARM updated or is a WARM
+ * tuple, there could be multiple index entries
+ * pointing to the root of this chain. We can't do
+ * index-only scans for such tuples without verifying
+ * index key check. So mark the page as !all_visible
+ */
+ if (HeapTupleHeaderIsWarmUpdated(tuple.t_data))
+ {
+ all_visible = false;
+ break;
+ }
+
/* Track newest xmin on page. */
if (TransactionIdFollows(xmin, visibility_cutoff_xid))
visibility_cutoff_xid = xmin;
@@ -1282,7 +1390,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* If any tuples need to be deleted, perform final vacuum cycle */
/* XXX put a threshold on min number of tuples here? */
- if (vacrelstats->num_dead_tuples > 0)
+ if (vacrelstats->num_dead_tuples > 0 || vacrelstats->num_warm_chains > 0)
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
@@ -1300,6 +1408,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
+ (vacrelstats->num_warm_chains > 0),
&indstats[i],
vacrelstats);
@@ -1367,7 +1476,10 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
*
* This routine marks dead tuples as unused and compacts out free
* space on their pages. Pages not having dead tuples recorded from
- * lazy_scan_heap are not visited at all.
+ * lazy_scan_heap are not visited at all. This routine also converts
+ * candidate WARM chains to HOT chains by clearing WARM related flags. The
+ * candidate chains are determined by the preceeding index scans after
+ * looking at the data collected by the first heap scan.
*
* Note: the reason for doing this as a second pass is we cannot remove
* the tuples until we've removed their index entries, and we want to
@@ -1376,7 +1488,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
static void
lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
{
- int tupindex;
+ int tupindex, chainindex;
int npages;
PGRUsage ru0;
Buffer vmbuffer = InvalidBuffer;
@@ -1385,33 +1497,69 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
npages = 0;
tupindex = 0;
- while (tupindex < vacrelstats->num_dead_tuples)
+ chainindex = 0;
+ while (tupindex < vacrelstats->num_dead_tuples ||
+ chainindex < vacrelstats->num_warm_chains)
{
- BlockNumber tblk;
+ BlockNumber tblk, chainblk, vacblk;
Buffer buf;
Page page;
Size freespace;
vacuum_delay_point();
- tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
- buf = ReadBufferExtended(onerel, MAIN_FORKNUM, tblk, RBM_NORMAL,
+ tblk = chainblk = InvalidBlockNumber;
+ if (chainindex < vacrelstats->num_warm_chains)
+ chainblk =
+ ItemPointerGetBlockNumber(&(vacrelstats->warm_chains[chainindex].chain_tid));
+
+ if (tupindex < vacrelstats->num_dead_tuples)
+ tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
+
+ if (tblk == InvalidBlockNumber)
+ vacblk = chainblk;
+ else if (chainblk == InvalidBlockNumber)
+ vacblk = tblk;
+ else
+ vacblk = Min(chainblk, tblk);
+
+ Assert(vacblk != InvalidBlockNumber);
+
+ buf = ReadBufferExtended(onerel, MAIN_FORKNUM, vacblk, RBM_NORMAL,
vac_strategy);
- if (!ConditionalLockBufferForCleanup(buf))
+
+
+ if (vacblk == chainblk)
+ LockBufferForCleanup(buf);
+ else if (!ConditionalLockBufferForCleanup(buf))
{
ReleaseBuffer(buf);
++tupindex;
continue;
}
- tupindex = lazy_vacuum_page(onerel, tblk, buf, tupindex, vacrelstats,
- &vmbuffer);
+
+ /*
+ * Convert WARM chains on this page. This should be done before
+ * vacuuming the page to ensure that we can correctly set visibility
+ * bits after clearing WARM chains.
+ *
+ * If we are going to vacuum this page then don't check for
+ * all-visibility just yet.
+ */
+ if (vacblk == chainblk)
+ chainindex = lazy_warmclear_page(onerel, chainblk, buf, chainindex,
+ vacrelstats, &vmbuffer, chainblk != tblk);
+
+ if (vacblk == tblk)
+ tupindex = lazy_vacuum_page(onerel, tblk, buf, tupindex, vacrelstats,
+ &vmbuffer);
/* Now that we've compacted the page, record its available space */
page = BufferGetPage(buf);
freespace = PageGetHeapFreeSpace(page);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, tblk, freespace);
+ RecordPageWithFreeSpace(onerel, vacblk, freespace);
npages++;
}
@@ -1430,6 +1578,107 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
}
/*
+ * lazy_warmclear_page() -- clear various WARM bits on the tuples.
+ *
+ * Caller must hold pin and buffer cleanup lock on the buffer.
+ *
+ * chainindex is the index in vacrelstats->warm_chains of the first dead
+ * tuple for this page. We assume the rest follow sequentially.
+ * The return value is the first tupindex after the tuples of this page.
+ *
+ * If check_all_visible is set then we also check if the page has now become
+ * all visible and update visibility map.
+ */
+static int
+lazy_warmclear_page(Relation onerel, BlockNumber blkno, Buffer buffer,
+ int chainindex, LVRelStats *vacrelstats, Buffer *vmbuffer,
+ bool check_all_visible)
+{
+ Page page = BufferGetPage(buffer);
+ OffsetNumber cleared_offnums[MaxHeapTuplesPerPage];
+ int num_cleared = 0;
+ TransactionId visibility_cutoff_xid;
+ bool all_frozen;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_WARMCLEARED, blkno);
+
+ START_CRIT_SECTION();
+
+ for (; chainindex < vacrelstats->num_warm_chains ; chainindex++)
+ {
+ BlockNumber tblk;
+ LVWarmChain *chain;
+
+ chain = &vacrelstats->warm_chains[chainindex];
+
+ tblk = ItemPointerGetBlockNumber(&chain->chain_tid);
+ if (tblk != blkno)
+ break; /* past end of tuples for this block */
+
+ /*
+ * Since a heap page can have no more than MaxHeapTuplesPerPage
+ * offnums and we process each offnum only once, MaxHeapTuplesPerPage
+ * size array should be enough to hold all cleared tuples in this page.
+ */
+ if (!chain->keep_warm_chain)
+ num_cleared += heap_clear_warm_chain(page, &chain->chain_tid,
+ cleared_offnums + num_cleared);
+ }
+
+ /*
+ * Mark buffer dirty before we write WAL.
+ */
+ MarkBufferDirty(buffer);
+
+ /* XLOG stuff */
+ if (RelationNeedsWAL(onerel))
+ {
+ XLogRecPtr recptr;
+
+ recptr = log_heap_warmclear(onerel, buffer,
+ cleared_offnums, num_cleared);
+ PageSetLSN(page, recptr);
+ }
+
+ END_CRIT_SECTION();
+
+ /* If not checking for all-visibility then we're done */
+ if (!check_all_visible)
+ return chainindex;
+
+ /*
+ * The following code should match the corresponding code in
+ * lazy_vacuum_page
+ **/
+ if (heap_page_is_all_visible(onerel, buffer, &visibility_cutoff_xid,
+ &all_frozen))
+ PageSetAllVisible(page);
+
+ /*
+ * All the changes to the heap page have been done. If the all-visible
+ * flag is now set, also set the VM all-visible bit (and, if possible, the
+ * all-frozen bit) unless this has already been done previously.
+ */
+ if (PageIsAllVisible(page))
+ {
+ uint8 vm_status = visibilitymap_get_status(onerel, blkno, vmbuffer);
+ uint8 flags = 0;
+
+ /* Set the VM all-frozen bit to flag, if needed */
+ if ((vm_status & VISIBILITYMAP_ALL_VISIBLE) == 0)
+ flags |= VISIBILITYMAP_ALL_VISIBLE;
+ if ((vm_status & VISIBILITYMAP_ALL_FROZEN) == 0 && all_frozen)
+ flags |= VISIBILITYMAP_ALL_FROZEN;
+
+ Assert(BufferIsValid(*vmbuffer));
+ if (flags != 0)
+ visibilitymap_set(onerel, blkno, buffer, InvalidXLogRecPtr,
+ *vmbuffer, visibility_cutoff_xid, flags);
+ }
+ return chainindex;
+}
+
+/*
* lazy_vacuum_page() -- free dead tuples on a page
* and repair its fragmentation.
*
@@ -1582,6 +1831,24 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Reset counters tracking number of WARM and CLEAR pointers per candidate TID.
+ * These counters are maintained per index and cleared when the next index is
+ * picked up for cleanup.
+ *
+ * We don't touch the keep_warm_chain since once a chain is known to be
+ * non-convertible, we must remember that across all indexes.
+ */
+static void
+lazy_reset_warm_pointer_count(LVRelStats *vacrelstats)
+{
+ int i;
+ for (i = 0; i < vacrelstats->num_warm_chains; i++)
+ {
+ LVWarmChain *chain = &vacrelstats->warm_chains[i];
+ chain->num_clear_pointers = chain->num_warm_pointers = 0;
+ }
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
@@ -1591,6 +1858,7 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
*/
static void
lazy_vacuum_index(Relation indrel,
+ bool clear_warm,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats)
{
@@ -1606,15 +1874,87 @@ lazy_vacuum_index(Relation indrel,
ivinfo.num_heap_tuples = vacrelstats->old_rel_tuples;
ivinfo.strategy = vac_strategy;
- /* Do bulk deletion */
- *stats = index_bulk_delete(&ivinfo, *stats,
- lazy_tid_reaped, (void *) vacrelstats);
+ /*
+ * If told, convert WARM chains into HOT chains.
+ *
+ * We must have already collected candidate WARM chains i.e. chains that
+ * have either all tuples with HEAP_WARM_TUPLE flag set or none.
+ *
+ * This works in two phases. In the first phase, we do a complete index
+ * scan and collect information about index pointers to the candidate
+ * chains, but we don't do conversion. To be precise, we count the number
+ * of WARM and CLEAR index pointers to each candidate chain and use that
+ * knowledge to arrive at a decision and do the actual conversion during
+ * the second phase (we kill known dead pointers though in this phase).
+ *
+ * In the second phase, for each candidate chain we check if we have seen a
+ * WARM index pointer. For such chains, we kill the CLEAR pointer and
+ * convert the WARM pointer into a CLEAR pointer. The heap tuples are
+ * cleared of WARM flags in the second heap scan. If we did not find any
+ * WARM pointer to a WARM chain, that means that the chain is reachable
+ * from the CLEAR pointer (because say WARM update did not add a new entry
+ * for this index). In that case, we do nothing. There is a third case
+ * where we find two CLEAR pointers to a candidate chain. This can happen
+ * because of aborted vacuums. We don't handle that case yet, but it should
+ * be possible to apply the same recheck logic and find which of the clear
+ * pointers is redundant and should be removed.
+ *
+ * For CLEAR chains, we just kill the WARM pointer, if it exist,s and keep
+ * the CLEAR pointer.
+ */
+ if (clear_warm)
+ {
+ /*
+ * Before starting the index scan, reset the counters of WARM and CLEAR
+ * pointers, probably carried forward from the previous index.
+ */
+ lazy_reset_warm_pointer_count(vacrelstats);
+
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_indexvac_phase1, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to remove %d row version, found "
+ "%0.f warm pointers, %0.f clear pointers, removed "
+ "%0.f warm pointers, removed %0.f clear pointers",
+ RelationGetRelationName(indrel),
+ vacrelstats->num_dead_tuples,
+ (*stats)->num_warm_pointers,
+ (*stats)->num_clear_pointers,
+ (*stats)->warm_pointers_removed,
+ (*stats)->clear_pointers_removed)));
+
+ (*stats)->num_warm_pointers = 0;
+ (*stats)->num_clear_pointers = 0;
+ (*stats)->warm_pointers_removed = 0;
+ (*stats)->clear_pointers_removed = 0;
+ (*stats)->pointers_cleared = 0;
+
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_indexvac_phase2, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to convert WARM pointers, found "
+ "%0.f WARM pointers, %0.f CLEAR pointers, removed "
+ "%0.f WARM pointers, removed %0.f CLEAR pointers, "
+ "cleared %0.f WARM pointers",
+ RelationGetRelationName(indrel),
+ (*stats)->num_warm_pointers,
+ (*stats)->num_clear_pointers,
+ (*stats)->warm_pointers_removed,
+ (*stats)->clear_pointers_removed,
+ (*stats)->pointers_cleared)));
+ }
+ else
+ {
+ /* Do bulk deletion */
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_tid_reaped, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to remove %d row versions",
+ RelationGetRelationName(indrel),
+ vacrelstats->num_dead_tuples),
+ errdetail("%s.", pg_rusage_show(&ru0))));
+ }
- ereport(elevel,
- (errmsg("scanned index \"%s\" to remove %d row versions",
- RelationGetRelationName(indrel),
- vacrelstats->num_dead_tuples),
- errdetail("%s.", pg_rusage_show(&ru0))));
}
/*
@@ -1988,9 +2328,11 @@ lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks)
if (vacrelstats->hasindex)
{
- maxtuples = (vac_work_mem * 1024L) / sizeof(ItemPointerData);
+ maxtuples = (vac_work_mem * 1024L) / (sizeof(ItemPointerData) +
+ sizeof(LVWarmChain));
maxtuples = Min(maxtuples, INT_MAX);
- maxtuples = Min(maxtuples, MaxAllocSize / sizeof(ItemPointerData));
+ maxtuples = Min(maxtuples, MaxAllocSize / (sizeof(ItemPointerData) +
+ sizeof(LVWarmChain)));
/* curious coding here to ensure the multiplication can't overflow */
if ((BlockNumber) (maxtuples / LAZY_ALLOC_TUPLES) > relblocks)
@@ -2008,6 +2350,57 @@ lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks)
vacrelstats->max_dead_tuples = (int) maxtuples;
vacrelstats->dead_tuples = (ItemPointer)
palloc(maxtuples * sizeof(ItemPointerData));
+
+ /*
+ * XXX Cheat for now and allocate the same size array for tracking warm
+ * chains. maxtuples must have been already adjusted above to ensure we
+ * don't cross vac_work_mem.
+ */
+ vacrelstats->num_warm_chains = 0;
+ vacrelstats->max_warm_chains = (int) maxtuples;
+ vacrelstats->warm_chains = (LVWarmChain *)
+ palloc0(maxtuples * sizeof(LVWarmChain));
+
+}
+
+/*
+ * lazy_record_clear_chain - remember one CLEAR chain
+ */
+static void
+lazy_record_clear_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr)
+{
+ /*
+ * The array shouldn't overflow under normal behavior, but perhaps it
+ * could if we are given a really small maintenance_work_mem. In that
+ * case, just forget the last few tuples (we'll get 'em next time).
+ */
+ if (vacrelstats->num_warm_chains < vacrelstats->max_warm_chains)
+ {
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].chain_tid = *itemptr;
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].is_postwarm_chain = 0;
+ vacrelstats->num_warm_chains++;
+ }
+}
+
+/*
+ * lazy_record_warm_chain - remember one WARM chain
+ */
+static void
+lazy_record_warm_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr)
+{
+ /*
+ * The array shouldn't overflow under normal behavior, but perhaps it
+ * could if we are given a really small maintenance_work_mem. In that
+ * case, just forget the last few tuples (we'll get 'em next time).
+ */
+ if (vacrelstats->num_warm_chains < vacrelstats->max_warm_chains)
+ {
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].chain_tid = *itemptr;
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].is_postwarm_chain = 1;
+ vacrelstats->num_warm_chains++;
+ }
}
/*
@@ -2038,8 +2431,8 @@ lazy_record_dead_tuple(LVRelStats *vacrelstats,
*
* Assumes dead_tuples array is in sorted order.
*/
-static bool
-lazy_tid_reaped(ItemPointer itemptr, void *state)
+static IndexBulkDeleteCallbackResult
+lazy_tid_reaped(ItemPointer itemptr, bool is_warm, void *state)
{
LVRelStats *vacrelstats = (LVRelStats *) state;
ItemPointer res;
@@ -2050,7 +2443,193 @@ lazy_tid_reaped(ItemPointer itemptr, void *state)
sizeof(ItemPointerData),
vac_cmp_itemptr);
- return (res != NULL);
+ return (res != NULL) ? IBDCR_DELETE : IBDCR_KEEP;
+}
+
+/*
+ * lazy_indexvac_phase1() -- run first pass of index vacuum
+ *
+ * This has the right signature to be an IndexBulkDeleteCallback.
+ */
+static IndexBulkDeleteCallbackResult
+lazy_indexvac_phase1(ItemPointer itemptr, bool is_warm, void *state)
+{
+ LVRelStats *vacrelstats = (LVRelStats *) state;
+ ItemPointer res;
+ LVWarmChain *chain;
+
+ res = (ItemPointer) bsearch((void *) itemptr,
+ (void *) vacrelstats->dead_tuples,
+ vacrelstats->num_dead_tuples,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+
+ if (res != NULL)
+ return IBDCR_DELETE;
+
+ chain = (LVWarmChain *) bsearch((void *) itemptr,
+ (void *) vacrelstats->warm_chains,
+ vacrelstats->num_warm_chains,
+ sizeof(LVWarmChain),
+ vac_cmp_warm_chain);
+ if (chain != NULL)
+ {
+ if (is_warm)
+ chain->num_warm_pointers++;
+ else
+ chain->num_clear_pointers++;
+ }
+ return IBDCR_KEEP;
+}
+
+/*
+ * lazy_indexvac_phase2() -- run first pass of index vacuum
+ *
+ * This has the right signature to be an IndexBulkDeleteCallback.
+ */
+static IndexBulkDeleteCallbackResult
+lazy_indexvac_phase2(ItemPointer itemptr, bool is_warm, void *state)
+{
+ LVRelStats *vacrelstats = (LVRelStats *) state;
+ LVWarmChain *chain;
+
+ chain = (LVWarmChain *) bsearch((void *) itemptr,
+ (void *) vacrelstats->warm_chains,
+ vacrelstats->num_warm_chains,
+ sizeof(LVWarmChain),
+ vac_cmp_warm_chain);
+
+ if (chain != NULL && (chain->keep_warm_chain != 1))
+ {
+ /*
+ * At no point, we can have more than 1 warm pointer to any chain and
+ * no more than 2 clear pointers.
+ */
+ Assert(chain->num_warm_pointers <= 1);
+ Assert(chain->num_clear_pointers <= 2);
+
+ if (chain->is_postwarm_chain == 1)
+ {
+ if (is_warm)
+ {
+ /*
+ * A WARM pointer, pointing to a WARM chain.
+ *
+ * Clear the warm pointer (and delete the CLEAR pointer). We
+ * may have already seen the CLEAR pointer in the scan and
+ * deleted that or we may see it later in the scan. It doesn't
+ * matter if we fail at any point because we won't clear up
+ * WARM bits on the heap tuples until we have dealt with the
+ * index pointers cleanly.
+ */
+ return IBDCR_CLEAR_WARM;
+ }
+ else
+ {
+ /*
+ * CLEAR pointer to a WARM chain.
+ */
+ if (chain->num_warm_pointers > 0)
+ {
+ /*
+ * If there exists a WARM pointer to the chain, we can
+ * delete the CLEAR pointer and clear the WARM bits on the
+ * heap tuples.
+ */
+ return IBDCR_DELETE;
+ }
+ else if (chain->num_clear_pointers == 1)
+ {
+ /*
+ * If this is the only pointer to a WARM chain, we must
+ * keep the CLEAR pointer.
+ *
+ * The presence of WARM chain indicates that the WARM update
+ * must have been committed good. But during the update
+ * this index was probably not updated and hence it
+ * contains just one, original CLEAR pointer to the chain.
+ * We should be able to clear the WARM bits on heap tuples
+ * unless we later find another index which prevents the
+ * cleanup.
+ */
+ return IBDCR_KEEP;
+ }
+ }
+ }
+ else
+ {
+ /*
+ * This is a CLEAR chain.
+ */
+ if (is_warm)
+ {
+ /*
+ * A WARM pointer to a CLEAR chain.
+ *
+ * This can happen when a WARM update is aborted. Later the HOT
+ * chain is pruned leaving behind only CLEAR tuples in the
+ * chain. But the WARM index pointer inserted in the index
+ * remains and it must now be deleted before we clear WARM bits
+ * from the heap tuple.
+ */
+ return IBDCR_DELETE;
+ }
+
+ /*
+ * CLEAR pointer to a CLEAR chain.
+ *
+ * If this is the only surviving CLEAR pointer, keep it and clear
+ * the WARM bits from the heap tuples.
+ */
+ if (chain->num_clear_pointers == 1)
+ return IBDCR_KEEP;
+
+ /*
+ * If there are more than 1 CLEAR pointers to this chain, we can
+ * apply the recheck logic and kill the redudant CLEAR pointer and
+ * convert the chain. But that's not yet done.
+ */
+ }
+
+ /*
+ * For everything else, we must keep the WARM bits and also keep the
+ * index pointers.
+ */
+ chain->keep_warm_chain = 1;
+ return IBDCR_KEEP;
+ }
+ return IBDCR_KEEP;
+}
+
+/*
+ * Comparator routines for use with qsort() and bsearch(). Similar to
+ * vac_cmp_itemptr, but right hand argument is LVWarmChain struct pointer.
+ */
+static int
+vac_cmp_warm_chain(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber(&((LVWarmChain *) right)->chain_tid);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber(&((LVWarmChain *) right)->chain_tid);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
}
/*
@@ -2166,6 +2745,18 @@ heap_page_is_all_visible(Relation rel, Buffer buf,
break;
}
+ /*
+ * If this or any other tuple in the chain ever WARM
+ * updated, there could be multiple index entries pointing
+ * to the root of this chain. We can't do index-only scans
+ * for such tuples without verifying index key check. So
+ * mark the page as !all_visible
+ */
+ if (HeapTupleHeaderIsWarmUpdated(tuple.t_data))
+ {
+ all_visible = false;
+ }
+
/* Track newest xmin on page. */
if (TransactionIdFollows(xmin, *visibility_cutoff_xid))
*visibility_cutoff_xid = xmin;
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 2142273..3e49a8f 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -270,6 +270,8 @@ ExecCloseIndices(ResultRelInfo *resultRelInfo)
List *
ExecInsertIndexTuples(TupleTableSlot *slot,
ItemPointer tupleid,
+ ItemPointer root_tid,
+ Bitmapset *modified_attrs,
EState *estate,
bool noDupErr,
bool *specConflict,
@@ -324,6 +326,17 @@ ExecInsertIndexTuples(TupleTableSlot *slot,
if (!indexInfo->ii_ReadyForInserts)
continue;
+ /*
+ * If modified_attrs is set, we only insert index entries for those
+ * indexes whose column has changed. All other indexes can use their
+ * existing index pointers to look up the new tuple
+ */
+ if (modified_attrs)
+ {
+ if (!bms_overlap(modified_attrs, indexInfo->ii_indxattrs))
+ continue;
+ }
+
/* Check for partial index */
if (indexInfo->ii_Predicate != NIL)
{
@@ -389,10 +402,11 @@ ExecInsertIndexTuples(TupleTableSlot *slot,
index_insert(indexRelation, /* index relation */
values, /* array of index Datums */
isnull, /* null flags */
- tupleid, /* tid of heap tuple */
+ root_tid, /* tid of heap or root tuple */
heapRelation, /* heap relation */
checkUnique, /* type of uniqueness check to do */
- indexInfo); /* index AM may need this */
+ indexInfo, /* index AM may need this */
+ (modified_attrs != NULL)); /* type of uniqueness check to do */
/*
* If the index has an associated exclusion constraint, check that.
@@ -791,6 +805,9 @@ retry:
{
if (!HeapTupleHeaderIsHeapLatest(tup->t_data, &tup->t_self))
HeapTupleHeaderGetNextTid(tup->t_data, &ctid_wait);
+ else
+ ItemPointerCopy(&tup->t_self, &ctid_wait);
+
reason_wait = indexInfo->ii_ExclusionOps ?
XLTW_RecheckExclusionConstr : XLTW_InsertIndex;
index_endscan(index_scan);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f20d728..747e4ce 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -399,6 +399,8 @@ ExecSimpleRelationInsert(EState *estate, TupleTableSlot *slot)
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self),
+ NULL,
estate, false, NULL,
NIL);
@@ -445,6 +447,8 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ bool warm_update;
+ Bitmapset *modified_attrs;
/* Check the constraints of the tuple */
if (rel->rd_att->constr)
@@ -455,13 +459,35 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
/* OK, update the tuple and index entries for it */
simple_heap_update(rel, &searchslot->tts_tuple->t_self,
- slot->tts_tuple);
+ slot->tts_tuple, &modified_attrs, &warm_update);
if (resultRelInfo->ri_NumIndices > 0 &&
- !HeapTupleIsHeapOnly(slot->tts_tuple))
+ (!HeapTupleIsHeapOnly(slot->tts_tuple) || warm_update))
+ {
+ ItemPointerData root_tid;
+
+ /*
+ * If we did a WARM update then we must index the tuple using its
+ * root line pointer and not the tuple TID itself.
+ */
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(tuple->t_self)),
+ HeapTupleHeaderGetRootOffset(tuple->t_data));
+ else
+ {
+ ItemPointerCopy(&tuple->t_self,
+ &root_tid);
+ bms_free(modified_attrs);
+ modified_attrs = NULL;
+ }
+
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &root_tid,
+ modified_attrs,
estate, false, NULL,
NIL);
+ }
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 2e9ff7d..f7bb6ca 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -39,6 +39,7 @@
#include "access/relscan.h"
#include "access/transam.h"
+#include "access/valid.h"
#include "executor/execdebug.h"
#include "executor/nodeBitmapHeapscan.h"
#include "pgstat.h"
@@ -395,11 +396,27 @@ bitgetpage(HeapScanDesc scan, TBMIterateResult *tbmres)
OffsetNumber offnum = tbmres->offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
+ bool recheck = false;
ItemPointerSet(&tid, page, offnum);
if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot,
- &heapTuple, NULL, true))
- scan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
+ &heapTuple, NULL, true, &recheck))
+ {
+ bool valid = true;
+
+ if (scan->rs_key)
+ HeapKeyTest(&heapTuple, RelationGetDescr(scan->rs_rd),
+ scan->rs_nkeys, scan->rs_key, valid);
+ if (valid)
+ scan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
+
+ /*
+ * If the heap tuple needs a recheck because of a WARM update,
+ * it's a lossy case.
+ */
+ if (recheck)
+ tbmres->recheck = true;
+ }
}
}
else
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index cb6aff9..dff4086 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -142,8 +142,8 @@ IndexNext(IndexScanState *node)
false); /* don't pfree */
/*
- * If the index was lossy, we have to recheck the index quals using
- * the fetched tuple.
+ * If the index was lossy or the tuple was WARM, we have to recheck
+ * the index quals using the fetched tuple.
*/
if (scandesc->xs_recheck)
{
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 95e1589..a1f3440 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -512,6 +512,7 @@ ExecInsert(ModifyTableState *mtstate,
/* insert index entries for tuple */
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self), NULL,
estate, true, &specConflict,
arbiterIndexes);
@@ -558,6 +559,7 @@ ExecInsert(ModifyTableState *mtstate,
/* insert index entries for tuple */
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self), NULL,
estate, false, NULL,
arbiterIndexes);
}
@@ -891,6 +893,9 @@ ExecUpdate(ItemPointer tupleid,
HTSU_Result result;
HeapUpdateFailureData hufd;
List *recheckIndexes = NIL;
+ Bitmapset *modified_attrs = NULL;
+ ItemPointerData root_tid;
+ bool warm_update;
/*
* abort the operation if not running transactions
@@ -1007,7 +1012,7 @@ lreplace:;
estate->es_output_cid,
estate->es_crosscheck_snapshot,
true /* wait for commit */ ,
- &hufd, &lockmode);
+ &hufd, &lockmode, &modified_attrs, &warm_update);
switch (result)
{
case HeapTupleSelfUpdated:
@@ -1094,10 +1099,28 @@ lreplace:;
* the t_self field.
*
* If it's a HOT update, we mustn't insert new index entries.
+ *
+ * If it's a WARM update, then we must insert new entries with TID
+ * pointing to the root of the WARM chain.
*/
- if (resultRelInfo->ri_NumIndices > 0 && !HeapTupleIsHeapOnly(tuple))
+ if (resultRelInfo->ri_NumIndices > 0 &&
+ (!HeapTupleIsHeapOnly(tuple) || warm_update))
+ {
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(tuple->t_self)),
+ HeapTupleHeaderGetRootOffset(tuple->t_data));
+ else
+ {
+ ItemPointerCopy(&tuple->t_self, &root_tid);
+ bms_free(modified_attrs);
+ modified_attrs = NULL;
+ }
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &root_tid,
+ modified_attrs,
estate, false, NULL, NIL);
+ }
}
if (canSetTag)
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 3a50488..806d812 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -1824,7 +1824,7 @@ pgstat_count_heap_insert(Relation rel, PgStat_Counter n)
* pgstat_count_heap_update - count a tuple update
*/
void
-pgstat_count_heap_update(Relation rel, bool hot)
+pgstat_count_heap_update(Relation rel, bool hot, bool warm)
{
PgStat_TableStatus *pgstat_info = rel->pgstat_info;
@@ -1842,6 +1842,8 @@ pgstat_count_heap_update(Relation rel, bool hot)
/* t_tuples_hot_updated is nontransactional, so just advance it */
if (hot)
pgstat_info->t_counts.t_tuples_hot_updated++;
+ else if (warm)
+ pgstat_info->t_counts.t_tuples_warm_updated++;
}
}
@@ -4324,6 +4326,7 @@ pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create)
result->tuples_updated = 0;
result->tuples_deleted = 0;
result->tuples_hot_updated = 0;
+ result->tuples_warm_updated = 0;
result->n_live_tuples = 0;
result->n_dead_tuples = 0;
result->changes_since_analyze = 0;
@@ -5433,6 +5436,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len)
tabentry->tuples_updated = tabmsg->t_counts.t_tuples_updated;
tabentry->tuples_deleted = tabmsg->t_counts.t_tuples_deleted;
tabentry->tuples_hot_updated = tabmsg->t_counts.t_tuples_hot_updated;
+ tabentry->tuples_warm_updated = tabmsg->t_counts.t_tuples_warm_updated;
tabentry->n_live_tuples = tabmsg->t_counts.t_delta_live_tuples;
tabentry->n_dead_tuples = tabmsg->t_counts.t_delta_dead_tuples;
tabentry->changes_since_analyze = tabmsg->t_counts.t_changed_tuples;
@@ -5460,6 +5464,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len)
tabentry->tuples_updated += tabmsg->t_counts.t_tuples_updated;
tabentry->tuples_deleted += tabmsg->t_counts.t_tuples_deleted;
tabentry->tuples_hot_updated += tabmsg->t_counts.t_tuples_hot_updated;
+ tabentry->tuples_warm_updated += tabmsg->t_counts.t_tuples_warm_updated;
/* If table was truncated, first reset the live/dead counters */
if (tabmsg->t_counts.t_truncated)
{
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5c13d26..7a9b48a 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -347,7 +347,7 @@ DecodeStandbyOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
static void
DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
- uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
+ uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP2_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
@@ -359,10 +359,6 @@ DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
switch (info)
{
- case XLOG_HEAP2_MULTI_INSERT:
- if (SnapBuildProcessChange(builder, xid, buf->origptr))
- DecodeMultiInsert(ctx, buf);
- break;
case XLOG_HEAP2_NEW_CID:
{
xl_heap_new_cid *xlrec;
@@ -390,6 +386,7 @@ DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_HEAP2_CLEANUP_INFO:
case XLOG_HEAP2_VISIBLE:
case XLOG_HEAP2_LOCK_UPDATED:
+ case XLOG_HEAP2_WARMCLEAR:
break;
default:
elog(ERROR, "unexpected RM_HEAP2_ID record type: %u", info);
@@ -418,6 +415,10 @@ DecodeHeapOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (SnapBuildProcessChange(builder, xid, buf->origptr))
DecodeInsert(ctx, buf);
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ if (SnapBuildProcessChange(builder, xid, buf->origptr))
+ DecodeMultiInsert(ctx, buf);
+ break;
/*
* Treat HOT update as normal updates. There is no useful
@@ -809,7 +810,7 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
/*
- * Decode XLOG_HEAP2_MULTI_INSERT_insert record into multiple tuplebufs.
+ * Decode XLOG_HEAP_MULTI_INSERT_insert record into multiple tuplebufs.
*
* Currently MULTI_INSERT will always contain the full tuples.
*/
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index a987d0d..b8677f3 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -145,6 +145,22 @@ pg_stat_get_tuples_hot_updated(PG_FUNCTION_ARGS)
Datum
+pg_stat_get_tuples_warm_updated(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ int64 result;
+ PgStat_StatTabEntry *tabentry;
+
+ if ((tabentry = pgstat_fetch_stat_tabentry(relid)) == NULL)
+ result = 0;
+ else
+ result = (int64) (tabentry->tuples_warm_updated);
+
+ PG_RETURN_INT64(result);
+}
+
+
+Datum
pg_stat_get_live_tuples(PG_FUNCTION_ARGS)
{
Oid relid = PG_GETARG_OID(0);
@@ -1644,6 +1660,21 @@ pg_stat_get_xact_tuples_hot_updated(PG_FUNCTION_ARGS)
}
Datum
+pg_stat_get_xact_tuples_warm_updated(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ int64 result;
+ PgStat_TableStatus *tabentry;
+
+ if ((tabentry = find_tabstat_entry(relid)) == NULL)
+ result = 0;
+ else
+ result = (int64) (tabentry->t_counts.t_tuples_warm_updated);
+
+ PG_RETURN_INT64(result);
+}
+
+Datum
pg_stat_get_xact_blocks_fetched(PG_FUNCTION_ARGS)
{
Oid relid = PG_GETARG_OID(0);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index ce55fc5..64dbaaa 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2338,6 +2338,7 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
list_free_deep(relation->rd_fkeylist);
list_free(relation->rd_indexlist);
bms_free(relation->rd_indexattr);
+ bms_free(relation->rd_exprindexattr);
bms_free(relation->rd_keyattr);
bms_free(relation->rd_pkattr);
bms_free(relation->rd_idattr);
@@ -4352,6 +4353,13 @@ RelationGetIndexList(Relation relation)
return list_copy(relation->rd_indexlist);
/*
+ * If the index list was invalidated, we better also invalidate the index
+ * attribute list (which should automatically invalidate other attributes
+ * such as primary key and replica identity)
+ */
+ relation->rd_indexattr = NULL;
+
+ /*
* We build the list we intend to return (in the caller's context) while
* doing the scan. After successfully completing the scan, we copy that
* list into the relcache entry. This avoids cache-context memory leakage
@@ -4759,15 +4767,19 @@ Bitmapset *
RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
{
Bitmapset *indexattrs; /* indexed columns */
+ Bitmapset *exprindexattrs; /* indexed columns in expression/prediacate
+ indexes */
Bitmapset *uindexattrs; /* columns in unique indexes */
Bitmapset *pkindexattrs; /* columns in the primary index */
Bitmapset *idindexattrs; /* columns in the replica identity */
+ Bitmapset *indxnotreadyattrs; /* columns in not ready indexes */
List *indexoidlist;
List *newindexoidlist;
Oid relpkindex;
Oid relreplindex;
ListCell *l;
MemoryContext oldcxt;
+ bool supportswarm = true;/* True if the table can be WARM updated */
/* Quick exit if we already computed the result. */
if (relation->rd_indexattr != NULL)
@@ -4782,6 +4794,10 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
return bms_copy(relation->rd_pkattr);
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
return bms_copy(relation->rd_idattr);
+ case INDEX_ATTR_BITMAP_EXPR_PREDICATE:
+ return bms_copy(relation->rd_exprindexattr);
+ case INDEX_ATTR_BITMAP_NOTREADY:
+ return bms_copy(relation->rd_indxnotreadyattr);
default:
elog(ERROR, "unknown attrKind %u", attrKind);
}
@@ -4822,9 +4838,11 @@ restart:
* won't be returned at all by RelationGetIndexList.
*/
indexattrs = NULL;
+ exprindexattrs = NULL;
uindexattrs = NULL;
pkindexattrs = NULL;
idindexattrs = NULL;
+ indxnotreadyattrs = NULL;
foreach(l, indexoidlist)
{
Oid indexOid = lfirst_oid(l);
@@ -4861,6 +4879,10 @@ restart:
indexattrs = bms_add_member(indexattrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
+ if (!indexInfo->ii_ReadyForInserts)
+ indxnotreadyattrs = bms_add_member(indxnotreadyattrs,
+ attrnum - FirstLowInvalidHeapAttributeNumber);
+
if (isKey)
uindexattrs = bms_add_member(uindexattrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
@@ -4876,10 +4898,29 @@ restart:
}
/* Collect all attributes used in expressions, too */
- pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &exprindexattrs);
/* Collect all attributes in the index predicate, too */
- pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &indexattrs);
+ pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &exprindexattrs);
+
+ /*
+ * indexattrs should include attributes referenced in index expressions
+ * and predicates too.
+ */
+ indexattrs = bms_add_members(indexattrs, exprindexattrs);
+
+ if (!indexInfo->ii_ReadyForInserts)
+ indxnotreadyattrs = bms_add_members(indxnotreadyattrs,
+ exprindexattrs);
+
+ /*
+ * Check if the index has amrecheck method defined. If the method is
+ * not defined, the index does not support WARM update. Completely
+ * disable WARM updates on such tables.
+ */
+ if (!indexDesc->rd_amroutine->amrecheck)
+ supportswarm = false;
+
index_close(indexDesc, AccessShareLock);
}
@@ -4912,15 +4953,22 @@ restart:
goto restart;
}
+ /* Remember if the table can do WARM updates */
+ relation->rd_supportswarm = supportswarm;
+
/* Don't leak the old values of these bitmaps, if any */
bms_free(relation->rd_indexattr);
relation->rd_indexattr = NULL;
+ bms_free(relation->rd_exprindexattr);
+ relation->rd_exprindexattr = NULL;
bms_free(relation->rd_keyattr);
relation->rd_keyattr = NULL;
bms_free(relation->rd_pkattr);
relation->rd_pkattr = NULL;
bms_free(relation->rd_idattr);
relation->rd_idattr = NULL;
+ bms_free(relation->rd_indxnotreadyattr);
+ relation->rd_indxnotreadyattr = NULL;
/*
* Now save copies of the bitmaps in the relcache entry. We intentionally
@@ -4933,7 +4981,9 @@ restart:
relation->rd_keyattr = bms_copy(uindexattrs);
relation->rd_pkattr = bms_copy(pkindexattrs);
relation->rd_idattr = bms_copy(idindexattrs);
- relation->rd_indexattr = bms_copy(indexattrs);
+ relation->rd_exprindexattr = bms_copy(exprindexattrs);
+ relation->rd_indexattr = bms_copy(bms_union(indexattrs, exprindexattrs));
+ relation->rd_indxnotreadyattr = bms_copy(indxnotreadyattrs);
MemoryContextSwitchTo(oldcxt);
/* We return our original working copy for caller to play with */
@@ -4947,6 +4997,10 @@ restart:
return bms_copy(relation->rd_pkattr);
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
return idindexattrs;
+ case INDEX_ATTR_BITMAP_EXPR_PREDICATE:
+ return exprindexattrs;
+ case INDEX_ATTR_BITMAP_NOTREADY:
+ return indxnotreadyattrs;
default:
elog(ERROR, "unknown attrKind %u", attrKind);
return NULL;
@@ -5559,6 +5613,7 @@ load_relcache_init_file(bool shared)
rel->rd_keyattr = NULL;
rel->rd_pkattr = NULL;
rel->rd_idattr = NULL;
+ rel->rd_indxnotreadyattr = NULL;
rel->rd_pubactions = NULL;
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
diff --git a/src/backend/utils/time/combocid.c b/src/backend/utils/time/combocid.c
index baff998..6a2e2f2 100644
--- a/src/backend/utils/time/combocid.c
+++ b/src/backend/utils/time/combocid.c
@@ -106,7 +106,7 @@ HeapTupleHeaderGetCmin(HeapTupleHeader tup)
{
CommandId cid = HeapTupleHeaderGetRawCommandId(tup);
- Assert(!(tup->t_infomask & HEAP_MOVED));
+ Assert(!(HeapTupleHeaderIsMoved(tup)));
Assert(TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tup)));
if (tup->t_infomask & HEAP_COMBOCID)
@@ -120,7 +120,7 @@ HeapTupleHeaderGetCmax(HeapTupleHeader tup)
{
CommandId cid = HeapTupleHeaderGetRawCommandId(tup);
- Assert(!(tup->t_infomask & HEAP_MOVED));
+ Assert(!(HeapTupleHeaderIsMoved(tup)));
/*
* Because GetUpdateXid() performs memory allocations if xmax is a
diff --git a/src/backend/utils/time/tqual.c b/src/backend/utils/time/tqual.c
index 519f3b6..e54d0df 100644
--- a/src/backend/utils/time/tqual.c
+++ b/src/backend/utils/time/tqual.c
@@ -186,7 +186,7 @@ HeapTupleSatisfiesSelf(HeapTuple htup, Snapshot snapshot, Buffer buffer)
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -205,7 +205,7 @@ HeapTupleSatisfiesSelf(HeapTuple htup, Snapshot snapshot, Buffer buffer)
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -377,7 +377,7 @@ HeapTupleSatisfiesToast(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -396,7 +396,7 @@ HeapTupleSatisfiesToast(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -471,7 +471,7 @@ HeapTupleSatisfiesUpdate(HeapTuple htup, CommandId curcid,
return HeapTupleInvisible;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -490,7 +490,7 @@ HeapTupleSatisfiesUpdate(HeapTuple htup, CommandId curcid,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -753,7 +753,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -772,7 +772,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -974,7 +974,7 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -993,7 +993,7 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -1180,7 +1180,7 @@ HeapTupleSatisfiesVacuum(HeapTuple htup, TransactionId OldestXmin,
if (HeapTupleHeaderXminInvalid(tuple))
return HEAPTUPLE_DEAD;
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_OFF)
+ else if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -1198,7 +1198,7 @@ HeapTupleSatisfiesVacuum(HeapTuple htup, TransactionId OldestXmin,
InvalidTransactionId);
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index f919cf8..8b7af1e 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -13,6 +13,7 @@
#define AMAPI_H
#include "access/genam.h"
+#include "access/itup.h"
/*
* We don't wish to include planner header files here, since most of an index
@@ -74,6 +75,14 @@ typedef bool (*aminsert_function) (Relation indexRelation,
Relation heapRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* insert this WARM tuple */
+typedef bool (*amwarminsert_function) (Relation indexRelation,
+ Datum *values,
+ bool *isnull,
+ ItemPointer heap_tid,
+ Relation heapRelation,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
@@ -152,6 +161,11 @@ typedef void (*aminitparallelscan_function) (void *target);
/* (re)start parallel index scan */
typedef void (*amparallelrescan_function) (IndexScanDesc scan);
+/* recheck index tuple and heap tuple match */
+typedef bool (*amrecheck_function) (Relation indexRel,
+ struct IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple);
+
/*
* API struct for an index AM. Note this must be stored in a single palloc'd
* chunk of memory.
@@ -198,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amwarminsert_function amwarminsert;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
@@ -217,6 +232,9 @@ typedef struct IndexAmRoutine
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
aminitparallelscan_function aminitparallelscan; /* can be NULL */
amparallelrescan_function amparallelrescan; /* can be NULL */
+
+ /* interface function to support WARM */
+ amrecheck_function amrecheck; /* can be NULL */
} IndexAmRoutine;
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index f467b18..965be45 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -75,12 +75,29 @@ typedef struct IndexBulkDeleteResult
bool estimated_count; /* num_index_tuples is an estimate */
double num_index_tuples; /* tuples remaining */
double tuples_removed; /* # removed during vacuum operation */
+ double num_warm_pointers; /* # WARM pointers found */
+ double num_clear_pointers; /* # CLEAR pointers found */
+ double pointers_cleared; /* # WARM pointers cleared */
+ double warm_pointers_removed; /* # WARM pointers removed */
+ double clear_pointers_removed; /* # CLEAR pointers removed */
BlockNumber pages_deleted; /* # unused pages in index */
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+/*
+ * IndexBulkDeleteCallback should return one of the following
+ */
+typedef enum IndexBulkDeleteCallbackResult
+{
+ IBDCR_KEEP, /* index tuple should be preserved */
+ IBDCR_DELETE, /* index tuple should be deleted */
+ IBDCR_CLEAR_WARM /* index tuple should be cleared of WARM bit */
+} IndexBulkDeleteCallbackResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
-typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
+typedef IndexBulkDeleteCallbackResult (*IndexBulkDeleteCallback) (
+ ItemPointer itemptr,
+ bool is_warm, void *state);
/* struct definitions appear in relscan.h */
typedef struct IndexScanDescData *IndexScanDesc;
@@ -135,7 +152,8 @@ extern bool index_insert(Relation indexRelation,
ItemPointer heap_t_ctid,
Relation heapRelation,
IndexUniqueCheck checkUnique,
- struct IndexInfo *indexInfo);
+ struct IndexInfo *indexInfo,
+ bool warm_update);
extern IndexScanDesc index_beginscan(Relation heapRelation,
Relation indexRelation,
diff --git a/src/include/access/hash.h b/src/include/access/hash.h
index eb1df57..f2094e3 100644
--- a/src/include/access/hash.h
+++ b/src/include/access/hash.h
@@ -281,6 +281,11 @@ typedef HashMetaPageData *HashMetaPage;
#define HASHPROC 1
#define HASHNProcs 1
+/*
+ * Flags overloaded on t_tid.ip_posid field. They are managed by
+ * ItemPointerSetFlags and corresponing routines.
+ */
+#define HASH_INDEX_WARM_POINTER 0x01
/* public routines */
@@ -291,6 +296,10 @@ extern bool hashinsert(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+extern bool hashwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
extern bool hashgettuple(IndexScanDesc scan, ScanDirection dir);
extern int64 hashgetbitmap(IndexScanDesc scan, TIDBitmap *tbm);
extern IndexScanDesc hashbeginscan(Relation rel, int nkeys, int norderbys);
@@ -360,6 +369,8 @@ extern void _hash_expandtable(Relation rel, Buffer metabuf);
extern void _hash_finish_split(Relation rel, Buffer metabuf, Buffer obuf,
Bucket obucket, uint32 maxbucket, uint32 highmask,
uint32 lowmask);
+extern void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems);
/* hashsearch.c */
extern bool _hash_next(IndexScanDesc scan, ScanDirection dir);
@@ -404,4 +415,8 @@ extern void hashbucketcleanup(Relation rel, Bucket cur_bucket,
bool bucket_has_garbage,
IndexBulkDeleteCallback callback, void *callback_state);
+/* hash.c */
+extern bool hashrecheck(Relation indexRel, struct IndexInfo *indexInfo,
+ IndexTuple indexTuple, Relation heapRel, HeapTuple heapTuple);
+
#endif /* HASH_H */
diff --git a/src/include/access/hash_xlog.h b/src/include/access/hash_xlog.h
index dfd9237..0549a5a 100644
--- a/src/include/access/hash_xlog.h
+++ b/src/include/access/hash_xlog.h
@@ -199,9 +199,10 @@ typedef struct xl_hash_delete
{
bool is_primary_bucket_page; /* TRUE if the operation is for
* primary bucket page */
+ uint16 nclearitems; /* # of items to clear of WARM bits */
} xl_hash_delete;
-#define SizeOfHashDelete (offsetof(xl_hash_delete, is_primary_bucket_page) + sizeof(bool))
+#define SizeOfHashDelete (offsetof(xl_hash_delete, nclearitems) + sizeof(uint16))
/*
* This is what we need for metapage update operation.
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 5540e12..2217af9 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -72,6 +72,20 @@ typedef struct HeapUpdateFailureData
CommandId cmax;
} HeapUpdateFailureData;
+typedef int HeapCheckWarmChainStatus;
+
+#define HCWC_CLEAR_TUPLE 0x0001
+#define HCWC_WARM_TUPLE 0x0002
+#define HCWC_WARM_UPDATED_TUPLE 0x0004
+
+#define HCWC_IS_MIXED(status) \
+ (((status) & (HCWC_CLEAR_TUPLE | HCWC_WARM_TUPLE)) != 0)
+#define HCWC_IS_ALL_WARM(status) \
+ (((status) & HCWC_CLEAR_TUPLE) == 0)
+#define HCWC_IS_ALL_CLEAR(status) \
+ (((status) & HCWC_WARM_TUPLE) == 0)
+#define HCWC_IS_WARM_UPDATED(status) \
+ (((status) & HCWC_WARM_UPDATED_TUPLE) != 0)
/* ----------------
* function prototypes for heap access method
@@ -137,9 +151,10 @@ extern bool heap_fetch(Relation relation, Snapshot snapshot,
Relation stats_relation);
extern bool heap_hot_search_buffer(ItemPointer tid, Relation relation,
Buffer buffer, Snapshot snapshot, HeapTuple heapTuple,
- bool *all_dead, bool first_call);
+ bool *all_dead, bool first_call, bool *recheck);
extern bool heap_hot_search(ItemPointer tid, Relation relation,
- Snapshot snapshot, bool *all_dead);
+ Snapshot snapshot, bool *all_dead,
+ bool *recheck, Buffer *buffer, HeapTuple heapTuple);
extern void heap_get_latest_tid(Relation relation, Snapshot snapshot,
ItemPointer tid);
@@ -161,7 +176,8 @@ extern void heap_abort_speculative(Relation relation, HeapTuple tuple);
extern HTSU_Result heap_update(Relation relation, ItemPointer otid,
HeapTuple newtup,
CommandId cid, Snapshot crosscheck, bool wait,
- HeapUpdateFailureData *hufd, LockTupleMode *lockmode);
+ HeapUpdateFailureData *hufd, LockTupleMode *lockmode,
+ Bitmapset **modified_attrsp, bool *warm_update);
extern HTSU_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy,
bool follow_update,
@@ -176,10 +192,16 @@ extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
extern Oid simple_heap_insert(Relation relation, HeapTuple tup);
extern void simple_heap_delete(Relation relation, ItemPointer tid);
extern void simple_heap_update(Relation relation, ItemPointer otid,
- HeapTuple tup);
+ HeapTuple tup,
+ Bitmapset **modified_attrs,
+ bool *warm_update);
extern void heap_sync(Relation relation);
extern void heap_update_snapshot(HeapScanDesc scan, Snapshot snapshot);
+extern HeapCheckWarmChainStatus heap_check_warm_chain(Page dp,
+ ItemPointer tid, bool stop_at_warm);
+extern int heap_clear_warm_chain(Page dp, ItemPointer tid,
+ OffsetNumber *cleared_offnums);
/* in heap/pruneheap.c */
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index e6019d5..66fd0ea 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -32,7 +32,7 @@
#define XLOG_HEAP_INSERT 0x00
#define XLOG_HEAP_DELETE 0x10
#define XLOG_HEAP_UPDATE 0x20
-/* 0x030 is free, was XLOG_HEAP_MOVE */
+#define XLOG_HEAP_MULTI_INSERT 0x30
#define XLOG_HEAP_HOT_UPDATE 0x40
#define XLOG_HEAP_CONFIRM 0x50
#define XLOG_HEAP_LOCK 0x60
@@ -47,18 +47,23 @@
/*
* We ran out of opcodes, so heapam.c now has a second RmgrId. These opcodes
* are associated with RM_HEAP2_ID, but are not logically different from
- * the ones above associated with RM_HEAP_ID. XLOG_HEAP_OPMASK applies to
- * these, too.
+ * the ones above associated with RM_HEAP_ID.
+ *
+ * In PG 10, we moved XLOG_HEAP2_MULTI_INSERT to RM_HEAP_ID. That allows us to
+ * use 0x80 bit in RM_HEAP2_ID, thus potentially creating another 8 possible
+ * opcodes in RM_HEAP2_ID.
*/
#define XLOG_HEAP2_REWRITE 0x00
#define XLOG_HEAP2_CLEAN 0x10
#define XLOG_HEAP2_FREEZE_PAGE 0x20
#define XLOG_HEAP2_CLEANUP_INFO 0x30
#define XLOG_HEAP2_VISIBLE 0x40
-#define XLOG_HEAP2_MULTI_INSERT 0x50
+#define XLOG_HEAP2_WARMCLEAR 0x50
#define XLOG_HEAP2_LOCK_UPDATED 0x60
#define XLOG_HEAP2_NEW_CID 0x70
+#define XLOG_HEAP2_OPMASK 0x70
+
/*
* xl_heap_insert/xl_heap_multi_insert flag values, 8 bits are available.
*/
@@ -80,6 +85,7 @@
#define XLH_UPDATE_CONTAINS_NEW_TUPLE (1<<4)
#define XLH_UPDATE_PREFIX_FROM_OLD (1<<5)
#define XLH_UPDATE_SUFFIX_FROM_OLD (1<<6)
+#define XLH_UPDATE_WARM_UPDATE (1<<7)
/* convenience macro for checking whether any form of old tuple was logged */
#define XLH_UPDATE_CONTAINS_OLD \
@@ -225,6 +231,14 @@ typedef struct xl_heap_clean
#define SizeOfHeapClean (offsetof(xl_heap_clean, ndead) + sizeof(uint16))
+typedef struct xl_heap_warmclear
+{
+ uint16 ncleared;
+ /* OFFSET NUMBERS are in the block reference 0 */
+} xl_heap_warmclear;
+
+#define SizeOfHeapWarmClear (offsetof(xl_heap_warmclear, ncleared) + sizeof(uint16))
+
/*
* Cleanup_info is required in some cases during a lazy VACUUM.
* Used for reporting the results of HeapTupleHeaderAdvanceLatestRemovedXid()
@@ -388,6 +402,8 @@ extern XLogRecPtr log_heap_clean(Relation reln, Buffer buffer,
OffsetNumber *nowdead, int ndead,
OffsetNumber *nowunused, int nunused,
TransactionId latestRemovedXid);
+extern XLogRecPtr log_heap_warmclear(Relation reln, Buffer buffer,
+ OffsetNumber *cleared, int ncleared);
extern XLogRecPtr log_heap_freeze(Relation reln, Buffer buffer,
TransactionId cutoff_xid, xl_heap_freeze_tuple *tuples,
int ntuples);
diff --git a/src/include/access/htup_details.h b/src/include/access/htup_details.h
index 4d614b7..bcefba6 100644
--- a/src/include/access/htup_details.h
+++ b/src/include/access/htup_details.h
@@ -201,6 +201,21 @@ struct HeapTupleHeaderData
* upgrade support */
#define HEAP_MOVED (HEAP_MOVED_OFF | HEAP_MOVED_IN)
+/*
+ * A WARM chain usually consists of two parts. Each of these parts are HOT
+ * chains in themselves i.e. all indexed columns has the same value, but a WARM
+ * update separates these parts. We need a mechanism to identify which part a
+ * tuple belongs to. We can't just look at if it's a
+ * HeapTupleHeaderIsWarmUpdated() because during WARM update, both old and new
+ * tuples are marked as WARM tuples.
+ *
+ * We need another infomask bit for this. But we use the same infomask bit that
+ * was earlier used for by old-style VACUUM FULL. This is safe because
+ * HEAP_WARM_TUPLE flag will always be set along with HEAP_WARM_UPDATED. So if
+ * HEAP_WARM_TUPLE and HEAP_WARM_UPDATED is set then we know that it's
+ * referring to red part of the WARM chain.
+ */
+#define HEAP_WARM_TUPLE 0x4000
#define HEAP_XACT_MASK 0xFFF0 /* visibility-related bits */
/*
@@ -260,7 +275,11 @@ struct HeapTupleHeaderData
* information stored in t_infomask2:
*/
#define HEAP_NATTS_MASK 0x07FF /* 11 bits for number of attributes */
-/* bits 0x0800 are available */
+#define HEAP_WARM_UPDATED 0x0800 /*
+ * This or a prior version of this
+ * tuple in the current HOT chain was
+ * once WARM updated
+ */
#define HEAP_LATEST_TUPLE 0x1000 /*
* This is the last tuple in chain and
* ip_posid points to the root line
@@ -271,7 +290,7 @@ struct HeapTupleHeaderData
#define HEAP_HOT_UPDATED 0x4000 /* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE 0x8000 /* this is heap-only tuple */
-#define HEAP2_XACT_MASK 0xF000 /* visibility-related bits */
+#define HEAP2_XACT_MASK 0xF800 /* visibility-related bits */
/*
@@ -396,7 +415,7 @@ struct HeapTupleHeaderData
/* SetCmin is reasonably simple since we never need a combo CID */
#define HeapTupleHeaderSetCmin(tup, cid) \
do { \
- Assert(!((tup)->t_infomask & HEAP_MOVED)); \
+ Assert(!HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
(tup)->t_infomask &= ~HEAP_COMBOCID; \
} while (0)
@@ -404,7 +423,7 @@ do { \
/* SetCmax must be used after HeapTupleHeaderAdjustCmax; see combocid.c */
#define HeapTupleHeaderSetCmax(tup, cid, iscombo) \
do { \
- Assert(!((tup)->t_infomask & HEAP_MOVED)); \
+ Assert(!HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
if (iscombo) \
(tup)->t_infomask |= HEAP_COMBOCID; \
@@ -414,7 +433,7 @@ do { \
#define HeapTupleHeaderGetXvac(tup) \
( \
- ((tup)->t_infomask & HEAP_MOVED) ? \
+ HeapTupleHeaderIsMoved(tup) ? \
(tup)->t_choice.t_heap.t_field3.t_xvac \
: \
InvalidTransactionId \
@@ -422,7 +441,7 @@ do { \
#define HeapTupleHeaderSetXvac(tup, xid) \
do { \
- Assert((tup)->t_infomask & HEAP_MOVED); \
+ Assert(HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_xvac = (xid); \
} while (0)
@@ -510,6 +529,21 @@ do { \
((tup)->t_infomask2 & HEAP_ONLY_TUPLE) != 0 \
)
+#define HeapTupleHeaderSetWarmUpdated(tup) \
+do { \
+ (tup)->t_infomask2 |= HEAP_WARM_UPDATED; \
+} while (0)
+
+#define HeapTupleHeaderClearWarmUpdated(tup) \
+do { \
+ (tup)->t_infomask2 &= ~HEAP_WARM_UPDATED; \
+} while (0)
+
+#define HeapTupleHeaderIsWarmUpdated(tup) \
+( \
+ ((tup)->t_infomask2 & HEAP_WARM_UPDATED) != 0 \
+)
+
/*
* Mark this as the last tuple in the HOT chain. Before PG v10 we used to store
* the TID of the tuple itself in t_ctid field to mark the end of the chain.
@@ -635,6 +669,58 @@ do { \
)
/*
+ * Macros to check if tuple is a moved-off/in tuple by VACUUM FULL in from
+ * pre-9.0 era. Such tuple must not have HEAP_WARM_TUPLE flag set.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderIsMovedOff(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED_OFF) \
+)
+
+#define HeapTupleHeaderIsMovedIn(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED_IN) \
+)
+
+#define HeapTupleHeaderIsMoved(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED) \
+)
+
+/*
+ * Check if tuple belongs to the second part of the WARM chain.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderIsWarm(tuple) \
+( \
+ HeapTupleHeaderIsWarmUpdated(tuple) && \
+ (((tuple)->t_infomask & HEAP_WARM_TUPLE) != 0) \
+)
+
+/*
+ * Mark tuple as a member of the second part of the chain. Must only be done on
+ * a tuple which is already marked a WARM-tuple.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderSetWarm(tuple) \
+( \
+ AssertMacro(HeapTupleHeaderIsWarmUpdated(tuple)), \
+ (tuple)->t_infomask |= HEAP_WARM_TUPLE \
+)
+
+#define HeapTupleHeaderClearWarm(tuple) \
+( \
+ (tuple)->t_infomask &= ~HEAP_WARM_TUPLE \
+)
+
+/*
* BITMAPLEN(NATTS) -
* Computes size of null bitmap given number of data columns.
*/
@@ -785,6 +871,24 @@ struct MinimalTupleData
#define HeapTupleClearHeapOnly(tuple) \
HeapTupleHeaderClearHeapOnly((tuple)->t_data)
+#define HeapTupleIsWarmUpdated(tuple) \
+ HeapTupleHeaderIsWarmUpdated((tuple)->t_data)
+
+#define HeapTupleSetWarmUpdated(tuple) \
+ HeapTupleHeaderSetWarmUpdated((tuple)->t_data)
+
+#define HeapTupleClearWarmUpdated(tuple) \
+ HeapTupleHeaderClearWarmUpdated((tuple)->t_data)
+
+#define HeapTupleIsWarm(tuple) \
+ HeapTupleHeaderIsWarm((tuple)->t_data)
+
+#define HeapTupleSetWarm(tuple) \
+ HeapTupleHeaderSetWarm((tuple)->t_data)
+
+#define HeapTupleClearWarm(tuple) \
+ HeapTupleHeaderClearWarm((tuple)->t_data)
+
#define HeapTupleGetOid(tuple) \
HeapTupleHeaderGetOid((tuple)->t_data)
diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h
index f9304db..163180d 100644
--- a/src/include/access/nbtree.h
+++ b/src/include/access/nbtree.h
@@ -427,6 +427,12 @@ typedef BTScanOpaqueData *BTScanOpaque;
#define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
/*
+ * Flags overloaded on t_tid.ip_posid field. They are managed by
+ * ItemPointerSetFlags and corresponing routines.
+ */
+#define BTREE_INDEX_WARM_POINTER 0x01
+
+/*
* external entry points for btree, in nbtree.c
*/
extern IndexBuildResult *btbuild(Relation heap, Relation index,
@@ -436,6 +442,10 @@ extern bool btinsert(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+extern bool btwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
extern IndexScanDesc btbeginscan(Relation rel, int nkeys, int norderbys);
extern Size btestimateparallelscan(void);
extern void btinitparallelscan(void *target);
@@ -487,10 +497,12 @@ extern void _bt_pageinit(Page page, Size size);
extern bool _bt_page_recyclable(Page page);
extern void _bt_delitems_delete(Relation rel, Buffer buf,
OffsetNumber *itemnos, int nitems, Relation heapRel);
-extern void _bt_delitems_vacuum(Relation rel, Buffer buf,
- OffsetNumber *itemnos, int nitems,
- BlockNumber lastBlockVacuumed);
+extern void _bt_handleitems_vacuum(Relation rel, Buffer buf,
+ OffsetNumber *delitemnos, int ndelitems,
+ OffsetNumber *clearitemnos, int nclearitems);
extern int _bt_pagedel(Relation rel, Buffer buf);
+extern void _bt_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems);
/*
* prototypes for functions in nbtsearch.c
@@ -537,6 +549,9 @@ extern bytea *btoptions(Datum reloptions, bool validate);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
+extern bool btrecheck(Relation indexRel, struct IndexInfo *indexInfo,
+ IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple);
/*
* prototypes for functions in nbtvalidate.c
diff --git a/src/include/access/nbtxlog.h b/src/include/access/nbtxlog.h
index d6a3085..7efd0d7 100644
--- a/src/include/access/nbtxlog.h
+++ b/src/include/access/nbtxlog.h
@@ -142,34 +142,20 @@ typedef struct xl_btree_reuse_page
/*
* This is what we need to know about vacuum of individual leaf index tuples.
* The WAL record can represent deletion of any number of index tuples on a
- * single index page when executed by VACUUM.
- *
- * For MVCC scans, lastBlockVacuumed will be set to InvalidBlockNumber.
- * For a non-MVCC index scans there is an additional correctness requirement
- * for applying these changes during recovery, which is that we must do one
- * of these two things for every block in the index:
- * * lock the block for cleanup and apply any required changes
- * * EnsureBlockUnpinned()
- * The purpose of this is to ensure that no index scans started before we
- * finish scanning the index are still running by the time we begin to remove
- * heap tuples.
- *
- * Any changes to any one block are registered on just one WAL record. All
- * blocks that we need to run EnsureBlockUnpinned() are listed as a block range
- * starting from the last block vacuumed through until this one. Individual
- * block numbers aren't given.
+ * single index page when executed by VACUUM. It also includes tuples which
+ * are cleared of WARM bits by VACUUM.
*
* Note that the *last* WAL record in any vacuum of an index is allowed to
* have a zero length array of offsets. Earlier records must have at least one.
*/
typedef struct xl_btree_vacuum
{
- BlockNumber lastBlockVacuumed;
-
- /* TARGET OFFSET NUMBERS FOLLOW */
+ uint16 ndelitems;
+ uint16 nclearitems;
+ /* ndelitems + nclearitems TARGET OFFSET NUMBERS FOLLOW */
} xl_btree_vacuum;
-#define SizeOfBtreeVacuum (offsetof(xl_btree_vacuum, lastBlockVacuumed) + sizeof(BlockNumber))
+#define SizeOfBtreeVacuum (offsetof(xl_btree_vacuum, nclearitems) + sizeof(uint16))
/*
* This is what we need to know about marking an empty branch for deletion.
diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h
index 3fc726d..fa178d3 100644
--- a/src/include/access/relscan.h
+++ b/src/include/access/relscan.h
@@ -104,6 +104,9 @@ typedef struct IndexScanDescData
/* index access method's private state */
void *opaque; /* access-method-specific info */
+ /* IndexInfo structure for this index */
+ struct IndexInfo *indexInfo;
+
/*
* In an index-only scan, a successful amgettuple call must fill either
* xs_itup (and xs_itupdesc) or xs_hitup (and xs_hitupdesc) to provide the
@@ -119,7 +122,7 @@ typedef struct IndexScanDescData
HeapTupleData xs_ctup; /* current heap tuple, if any */
Buffer xs_cbuf; /* current heap buffer in scan, if any */
/* NB: if xs_cbuf is not InvalidBuffer, we hold a pin on that buffer */
- bool xs_recheck; /* T means scan keys must be rechecked */
+ bool xs_recheck; /* T means scan keys must be rechecked for each tuple */
/*
* When fetching with an ordering operator, the values of the ORDER BY
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index 20bec90..f92ec29 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -89,6 +89,13 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
Datum *values,
bool *isnull);
+extern void FormIndexPlainDatum(IndexInfo *indexInfo,
+ Relation heapRel,
+ HeapTuple heapTup,
+ Datum *values,
+ bool *isnull,
+ bool *isavail);
+
extern void index_build(Relation heapRelation,
Relation indexRelation,
IndexInfo *indexInfo,
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 836d6ff..0ca6e22 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2769,6 +2769,8 @@ DATA(insert OID = 1933 ( pg_stat_get_tuples_deleted PGNSP PGUID 12 1 0 0 0 f f
DESCR("statistics: number of tuples deleted");
DATA(insert OID = 1972 ( pg_stat_get_tuples_hot_updated PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_hot_updated _null_ _null_ _null_ ));
DESCR("statistics: number of tuples hot updated");
+DATA(insert OID = 3355 ( pg_stat_get_tuples_warm_updated PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_warm_updated _null_ _null_ _null_ ));
+DESCR("statistics: number of tuples warm updated");
DATA(insert OID = 2878 ( pg_stat_get_live_tuples PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_live_tuples _null_ _null_ _null_ ));
DESCR("statistics: number of live tuples");
DATA(insert OID = 2879 ( pg_stat_get_dead_tuples PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_dead_tuples _null_ _null_ _null_ ));
@@ -2921,6 +2923,8 @@ DATA(insert OID = 3042 ( pg_stat_get_xact_tuples_deleted PGNSP PGUID 12 1 0 0
DESCR("statistics: number of tuples deleted in current transaction");
DATA(insert OID = 3043 ( pg_stat_get_xact_tuples_hot_updated PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_tuples_hot_updated _null_ _null_ _null_ ));
DESCR("statistics: number of tuples hot updated in current transaction");
+DATA(insert OID = 3356 ( pg_stat_get_xact_tuples_warm_updated PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_tuples_warm_updated _null_ _null_ _null_ ));
+DESCR("statistics: number of tuples warm updated in current transaction");
DATA(insert OID = 3044 ( pg_stat_get_xact_blocks_fetched PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_blocks_fetched _null_ _null_ _null_ ));
DESCR("statistics: number of blocks fetched in current transaction");
DATA(insert OID = 3045 ( pg_stat_get_xact_blocks_hit PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_blocks_hit _null_ _null_ _null_ ));
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9472ecc..b355b61 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -25,6 +25,7 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_HEAP_BLKS_WARMCLEARED 7
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 02dbe7b..c4495a3 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -382,6 +382,7 @@ extern void UnregisterExprContextCallback(ExprContext *econtext,
extern void ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative);
extern void ExecCloseIndices(ResultRelInfo *resultRelInfo);
extern List *ExecInsertIndexTuples(TupleTableSlot *slot, ItemPointer tupleid,
+ ItemPointer root_tid, Bitmapset *modified_attrs,
EState *estate, bool noDupErr, bool *specConflict,
List *arbiterIndexes);
extern bool ExecCheckIndexConstraints(TupleTableSlot *slot, EState *estate,
diff --git a/src/include/executor/nodeIndexscan.h b/src/include/executor/nodeIndexscan.h
index ea3f3a5..ebeec74 100644
--- a/src/include/executor/nodeIndexscan.h
+++ b/src/include/executor/nodeIndexscan.h
@@ -41,5 +41,4 @@ extern void ExecIndexEvalRuntimeKeys(ExprContext *econtext,
extern bool ExecIndexEvalArrayKeys(ExprContext *econtext,
IndexArrayKeyInfo *arrayKeys, int numArrayKeys);
extern bool ExecIndexAdvanceArrayKeys(IndexArrayKeyInfo *arrayKeys, int numArrayKeys);
-
#endif /* NODEINDEXSCAN_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index f856f60..cd09553 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -66,6 +66,7 @@ typedef struct IndexInfo
NodeTag type;
int ii_NumIndexAttrs;
AttrNumber ii_KeyAttrNumbers[INDEX_MAX_KEYS];
+ Bitmapset *ii_indxattrs; /* bitmap of all columns used in this index */
List *ii_Expressions; /* list of Expr */
List *ii_ExpressionsState; /* list of ExprState */
List *ii_Predicate; /* list of Expr */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2daf32..af8a3ba 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -105,6 +105,7 @@ typedef struct PgStat_TableCounts
PgStat_Counter t_tuples_updated;
PgStat_Counter t_tuples_deleted;
PgStat_Counter t_tuples_hot_updated;
+ PgStat_Counter t_tuples_warm_updated;
bool t_truncated;
PgStat_Counter t_delta_live_tuples;
@@ -625,6 +626,7 @@ typedef struct PgStat_StatTabEntry
PgStat_Counter tuples_updated;
PgStat_Counter tuples_deleted;
PgStat_Counter tuples_hot_updated;
+ PgStat_Counter tuples_warm_updated;
PgStat_Counter n_live_tuples;
PgStat_Counter n_dead_tuples;
@@ -1257,7 +1259,7 @@ pgstat_report_wait_end(void)
(pgStatBlockWriteTime += (n))
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
-extern void pgstat_count_heap_update(Relation rel, bool hot);
+extern void pgstat_count_heap_update(Relation rel, bool hot, bool warm);
extern void pgstat_count_heap_delete(Relation rel);
extern void pgstat_count_truncate(Relation rel);
extern void pgstat_update_heap_dead_tuples(Relation rel, int delta);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index a617a7c..fbac7c0 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -138,9 +138,14 @@ typedef struct RelationData
/* data managed by RelationGetIndexAttrBitmap: */
Bitmapset *rd_indexattr; /* identifies columns used in indexes */
+ Bitmapset *rd_exprindexattr; /* indentified columns used in expression or
+ predicate indexes */
+ Bitmapset *rd_indxnotreadyattr; /* columns used by indexes not yet
+ ready */
Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */
Bitmapset *rd_pkattr; /* cols included in primary key */
Bitmapset *rd_idattr; /* included in replica identity index */
+ bool rd_supportswarm;/* True if the table can be WARM updated */
PublicationActions *rd_pubactions; /* publication actions */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index da36b67..d18bd09 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -50,7 +50,9 @@ typedef enum IndexAttrBitmapKind
INDEX_ATTR_BITMAP_ALL,
INDEX_ATTR_BITMAP_KEY,
INDEX_ATTR_BITMAP_PRIMARY_KEY,
- INDEX_ATTR_BITMAP_IDENTITY_KEY
+ INDEX_ATTR_BITMAP_IDENTITY_KEY,
+ INDEX_ATTR_BITMAP_EXPR_PREDICATE,
+ INDEX_ATTR_BITMAP_NOTREADY
} IndexAttrBitmapKind;
extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..347c4ce 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -15,6 +15,7 @@ SUBDIRS = \
test_pg_dump \
test_rls_hooks \
test_shm_mq \
+ warm \
worker_spi
all: submake-generated-headers
diff --git a/src/test/regress/expected/alter_generic.out b/src/test/regress/expected/alter_generic.out
index b01be59..37719c9 100644
--- a/src/test/regress/expected/alter_generic.out
+++ b/src/test/regress/expected/alter_generic.out
@@ -161,15 +161,15 @@ ALTER SERVER alt_fserv1 RENAME TO alt_fserv3; -- OK
SELECT fdwname FROM pg_foreign_data_wrapper WHERE fdwname like 'alt_fdw%';
fdwname
----------
- alt_fdw2
alt_fdw3
+ alt_fdw2
(2 rows)
SELECT srvname FROM pg_foreign_server WHERE srvname like 'alt_fserv%';
srvname
------------
- alt_fserv2
alt_fserv3
+ alt_fserv2
(2 rows)
--
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index bd13ae6..44c59ae 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1732,6 +1732,7 @@ pg_stat_all_tables| SELECT c.oid AS relid,
pg_stat_get_tuples_updated(c.oid) AS n_tup_upd,
pg_stat_get_tuples_deleted(c.oid) AS n_tup_del,
pg_stat_get_tuples_hot_updated(c.oid) AS n_tup_hot_upd,
+ pg_stat_get_tuples_warm_updated(c.oid) AS n_tup_warm_upd,
pg_stat_get_live_tuples(c.oid) AS n_live_tup,
pg_stat_get_dead_tuples(c.oid) AS n_dead_tup,
pg_stat_get_mod_since_analyze(c.oid) AS n_mod_since_analyze,
@@ -1875,6 +1876,7 @@ pg_stat_sys_tables| SELECT pg_stat_all_tables.relid,
pg_stat_all_tables.n_tup_upd,
pg_stat_all_tables.n_tup_del,
pg_stat_all_tables.n_tup_hot_upd,
+ pg_stat_all_tables.n_tup_warm_upd,
pg_stat_all_tables.n_live_tup,
pg_stat_all_tables.n_dead_tup,
pg_stat_all_tables.n_mod_since_analyze,
@@ -1918,6 +1920,7 @@ pg_stat_user_tables| SELECT pg_stat_all_tables.relid,
pg_stat_all_tables.n_tup_upd,
pg_stat_all_tables.n_tup_del,
pg_stat_all_tables.n_tup_hot_upd,
+ pg_stat_all_tables.n_tup_warm_upd,
pg_stat_all_tables.n_live_tup,
pg_stat_all_tables.n_dead_tup,
pg_stat_all_tables.n_mod_since_analyze,
@@ -1955,7 +1958,8 @@ pg_stat_xact_all_tables| SELECT c.oid AS relid,
pg_stat_get_xact_tuples_inserted(c.oid) AS n_tup_ins,
pg_stat_get_xact_tuples_updated(c.oid) AS n_tup_upd,
pg_stat_get_xact_tuples_deleted(c.oid) AS n_tup_del,
- pg_stat_get_xact_tuples_hot_updated(c.oid) AS n_tup_hot_upd
+ pg_stat_get_xact_tuples_hot_updated(c.oid) AS n_tup_hot_upd,
+ pg_stat_get_xact_tuples_warm_updated(c.oid) AS n_tup_warm_upd
FROM ((pg_class c
LEFT JOIN pg_index i ON ((c.oid = i.indrelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
@@ -1971,7 +1975,8 @@ pg_stat_xact_sys_tables| SELECT pg_stat_xact_all_tables.relid,
pg_stat_xact_all_tables.n_tup_ins,
pg_stat_xact_all_tables.n_tup_upd,
pg_stat_xact_all_tables.n_tup_del,
- pg_stat_xact_all_tables.n_tup_hot_upd
+ pg_stat_xact_all_tables.n_tup_hot_upd,
+ pg_stat_xact_all_tables.n_tup_warm_upd
FROM pg_stat_xact_all_tables
WHERE ((pg_stat_xact_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_stat_xact_all_tables.schemaname ~ '^pg_toast'::text));
pg_stat_xact_user_functions| SELECT p.oid AS funcid,
@@ -1993,7 +1998,8 @@ pg_stat_xact_user_tables| SELECT pg_stat_xact_all_tables.relid,
pg_stat_xact_all_tables.n_tup_ins,
pg_stat_xact_all_tables.n_tup_upd,
pg_stat_xact_all_tables.n_tup_del,
- pg_stat_xact_all_tables.n_tup_hot_upd
+ pg_stat_xact_all_tables.n_tup_hot_upd,
+ pg_stat_xact_all_tables.n_tup_warm_upd
FROM pg_stat_xact_all_tables
WHERE ((pg_stat_xact_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_stat_xact_all_tables.schemaname !~ '^pg_toast'::text));
pg_statio_all_indexes| SELECT c.oid AS relid,
diff --git a/src/test/regress/expected/warm.out b/src/test/regress/expected/warm.out
new file mode 100644
index 0000000..6391891
--- /dev/null
+++ b/src/test/regress/expected/warm.out
@@ -0,0 +1,367 @@
+CREATE TABLE updtst_tab1 (a integer unique, b int, c text, d text);
+CREATE INDEX updtst_indx1 ON updtst_tab1 (b);
+INSERT INTO updtst_tab1
+ SELECT generate_series(1,10000), generate_series(70001, 80000), 'foo', 'bar';
+-- This should be a HOT update as non-index key is updated, but the
+-- page won't have any free space, so probably a non-HOT update
+UPDATE updtst_tab1 SET c = 'foo1' WHERE a = 1;
+-- Next update should be a HOT update as dead space is recycled
+UPDATE updtst_tab1 SET c = 'foo2' WHERE a = 1;
+-- And next too
+UPDATE updtst_tab1 SET c = 'foo3' WHERE a = 1;
+-- Now update one of the index key columns
+UPDATE updtst_tab1 SET b = b + 70000 WHERE a = 1;
+-- Ensure that the correct row is fetched
+SELECT * FROM updtst_tab1 WHERE a = 1;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+-- Even when seqscan is disabled and indexscan is forced
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab1
+ Recheck Cond: (b = 140001)
+ -> Bitmap Index Scan on updtst_indx1
+ Index Cond: (b = 140001)
+(4 rows)
+
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+-- Check if index only scan works correctly
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab1
+ Recheck Cond: (b = 140001)
+ -> Bitmap Index Scan on updtst_indx1
+ Index Cond: (b = 140001)
+(4 rows)
+
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ b
+--------
+ 140001
+(1 row)
+
+-- Table must be vacuumed to force index-only scan
+VACUUM updtst_tab1;
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx1 on updtst_tab1
+ Index Cond: (b = 140001)
+(2 rows)
+
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ b
+--------
+ 140001
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab1;
+------------------
+CREATE TABLE updtst_tab2 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx2 ON updtst_tab2 (b);
+INSERT INTO updtst_tab2
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+UPDATE updtst_tab2 SET b = b + 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo1' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab2 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo6' WHERE a = 1;
+SELECT count(*) FROM updtst_tab2 WHERE c = 'foo';
+ count
+-------
+ 99
+(1 row)
+
+SELECT * FROM updtst_tab2 WHERE c = 'foo6';
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab2
+ Recheck Cond: (b = 701)
+ -> Bitmap Index Scan on updtst_indx2
+ Index Cond: (b = 701)
+(4 rows)
+
+SELECT * FROM updtst_tab2 WHERE a = 1;
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab2
+ Recheck Cond: (b = 701)
+ -> Bitmap Index Scan on updtst_indx2
+ Index Cond: (b = 701)
+(4 rows)
+
+SELECT * FROM updtst_tab2 WHERE b = 701;
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+VACUUM updtst_tab2;
+EXPLAIN (costs off) SELECT b FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx2 on updtst_tab2
+ Index Cond: (b = 701)
+(2 rows)
+
+SELECT b FROM updtst_tab2 WHERE b = 701;
+ b
+-----
+ 701
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab2;
+------------------
+CREATE TABLE updtst_tab3 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx3 ON updtst_tab3 (b);
+INSERT INTO updtst_tab3
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo1', b = b + 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo6' WHERE a = 1;
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo11', b = b + 750 WHERE b = 701;
+UPDATE updtst_tab3 SET c = 'foo12' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 1;
+COMMIT;
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+ count
+-------
+ 99
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo6';
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo12';
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE a = 1;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 701;
+ QUERY PLAN
+-------------------------
+ Seq Scan on updtst_tab3
+ Filter: (b = 701)
+(2 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 701;
+ b
+---
+(0 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 1421;
+ b
+------
+ 1421
+(1 row)
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo23' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo24' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo25' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo26' WHERE a = 2;
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+SET enable_seqscan = false;
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 750 WHERE b = 702;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 2;
+COMMIT;
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+ count
+-------
+ 98
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo26';
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo22';
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 702;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE a = 2;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+-- Try fetching both old and new value using updtst_indx3
+SELECT * FROM updtst_tab3 WHERE b = 702;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 702;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx3 on updtst_tab3
+ Index Cond: (b = 702)
+(2 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 702;
+ b
+---
+(0 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 1422;
+ b
+------
+ 1422
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab3;
+------------------
+CREATE TABLE test_warm (a text unique, b text);
+CREATE INDEX test_warmindx ON test_warm (lower(a));
+INSERT INTO test_warm values ('test', 'foo');
+UPDATE test_warm SET a = 'TEST';
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+explain select * from test_warm where lower(a) = 'test';
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Bitmap Heap Scan on test_warm (cost=4.18..12.65 rows=4 width=64)
+ Recheck Cond: (lower(a) = 'test'::text)
+ -> Bitmap Index Scan on test_warmindx (cost=0.00..4.18 rows=4 width=0)
+ Index Cond: (lower(a) = 'test'::text)
+(4 rows)
+
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+select *, ctid from test_warm where a = 'test';
+ a | b | ctid
+---+---+------
+(0 rows)
+
+select *, ctid from test_warm where a = 'TEST';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+set enable_bitmapscan TO false;
+explain select * from test_warm where lower(a) = 'test';
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Index Scan using test_warmindx on test_warm (cost=0.15..20.22 rows=4 width=64)
+ Index Cond: (lower(a) = 'test'::text)
+(2 rows)
+
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+DROP TABLE test_warm;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index ea7b5b4..7cc0d21 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -42,6 +42,8 @@ test: create_type
test: create_table
test: create_function_2
+test: warm
+
# ----------
# Load huge amounts of data
# We should split the data files into single files and then
diff --git a/src/test/regress/sql/warm.sql b/src/test/regress/sql/warm.sql
new file mode 100644
index 0000000..3a078dd
--- /dev/null
+++ b/src/test/regress/sql/warm.sql
@@ -0,0 +1,170 @@
+
+CREATE TABLE updtst_tab1 (a integer unique, b int, c text, d text);
+CREATE INDEX updtst_indx1 ON updtst_tab1 (b);
+INSERT INTO updtst_tab1
+ SELECT generate_series(1,10000), generate_series(70001, 80000), 'foo', 'bar';
+
+-- This should be a HOT update as non-index key is updated, but the
+-- page won't have any free space, so probably a non-HOT update
+UPDATE updtst_tab1 SET c = 'foo1' WHERE a = 1;
+
+-- Next update should be a HOT update as dead space is recycled
+UPDATE updtst_tab1 SET c = 'foo2' WHERE a = 1;
+
+-- And next too
+UPDATE updtst_tab1 SET c = 'foo3' WHERE a = 1;
+
+-- Now update one of the index key columns
+UPDATE updtst_tab1 SET b = b + 70000 WHERE a = 1;
+
+-- Ensure that the correct row is fetched
+SELECT * FROM updtst_tab1 WHERE a = 1;
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Even when seqscan is disabled and indexscan is forced
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Check if index only scan works correctly
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Table must be vacuumed to force index-only scan
+VACUUM updtst_tab1;
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab1;
+
+------------------
+
+CREATE TABLE updtst_tab2 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx2 ON updtst_tab2 (b);
+INSERT INTO updtst_tab2
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+
+UPDATE updtst_tab2 SET b = b + 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo1' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab2 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo6' WHERE a = 1;
+
+SELECT count(*) FROM updtst_tab2 WHERE c = 'foo';
+SELECT * FROM updtst_tab2 WHERE c = 'foo6';
+
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+SELECT * FROM updtst_tab2 WHERE a = 1;
+
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+SELECT * FROM updtst_tab2 WHERE b = 701;
+
+VACUUM updtst_tab2;
+EXPLAIN (costs off) SELECT b FROM updtst_tab2 WHERE b = 701;
+SELECT b FROM updtst_tab2 WHERE b = 701;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab2;
+------------------
+
+CREATE TABLE updtst_tab3 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx3 ON updtst_tab3 (b);
+INSERT INTO updtst_tab3
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo1', b = b + 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo6' WHERE a = 1;
+
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo11', b = b + 750 WHERE b = 701;
+UPDATE updtst_tab3 SET c = 'foo12' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 1;
+COMMIT;
+
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+SELECT * FROM updtst_tab3 WHERE c = 'foo6';
+SELECT * FROM updtst_tab3 WHERE c = 'foo12';
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+SELECT * FROM updtst_tab3 WHERE a = 1;
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 701;
+SELECT b FROM updtst_tab3 WHERE b = 701;
+SELECT b FROM updtst_tab3 WHERE b = 1421;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo23' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo24' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo25' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo26' WHERE a = 2;
+
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+
+SET enable_seqscan = false;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 750 WHERE b = 702;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 2;
+COMMIT;
+
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+SELECT * FROM updtst_tab3 WHERE c = 'foo26';
+SELECT * FROM updtst_tab3 WHERE c = 'foo22';
+
+SELECT * FROM updtst_tab3 WHERE b = 702;
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+SELECT * FROM updtst_tab3 WHERE a = 2;
+
+-- Try fetching both old and new value using updtst_indx3
+SELECT * FROM updtst_tab3 WHERE b = 702;
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 702;
+SELECT b FROM updtst_tab3 WHERE b = 702;
+SELECT b FROM updtst_tab3 WHERE b = 1422;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab3;
+------------------
+
+CREATE TABLE test_warm (a text unique, b text);
+CREATE INDEX test_warmindx ON test_warm (lower(a));
+INSERT INTO test_warm values ('test', 'foo');
+UPDATE test_warm SET a = 'TEST';
+select *, ctid from test_warm where lower(a) = 'test';
+explain select * from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where a = 'test';
+select *, ctid from test_warm where a = 'TEST';
+set enable_bitmapscan TO false;
+explain select * from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where lower(a) = 'test';
+DROP TABLE test_warm;
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 13:25 Robert Haas <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 3 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-21 13:25 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee
<[email protected]> wrote:
>> Yeah. So what's the deal with this? Is somebody working on figuring
>> out a different approach that would reduce this overhead? Are we
>> going to defer WARM to v11? Or is the intent to just ignore the 5-10%
>> slowdown on a single column update and commit everything anyway?
>
> I think I should clarify something. The test case does a single column
> update, but it also has columns which are very wide, has an index on many
> columns (and it updates a column early in the list). In addition, in the
> test Mithun updated all 10million rows of the table in a single transaction,
> used UNLOGGED table and fsync was turned off.
>
> TBH I see many artificial scenarios here. It will be very useful if he can
> rerun the query with some of these restrictions lifted. I'm all for
> addressing whatever we can, but I am not sure if this test demonstrates a
> real world usage.
That's a very fair point, but if these patches - or some of them - are
going to get committed then these things need to get discussed. Let's
not just have nothing-nothing-nothing giant unagreed code drop.
I think that very wide columns and highly indexed tables are not
particularly unrealistic, nor do I think updating all the rows is
particularly unrealistic. Sure, it's not everything, but it's
something. Now, I would agree that all of that PLUS unlogged tables
with fsync=off is not too realistic. What kind of regression would we
observe if we eliminated those last two variables?
> Having said that, may be if we can do a few things to reduce the overhead.
>
> - Check if the page has enough free space to perform a HOT/WARM update. If
> not, don't look for all index keys.
> - Pass bitmaps separately for each index and bail out early if we conclude
> neither HOT nor WARM is possible. In this case since there is just one index
> and as soon as we check the second column we know neither HOT nor WARM is
> possible, we will return early. It might complicate the API a lot, but I can
> give it a shot if that's what is needed to make progress.
I think that whether the code ends up getting contorted is an
important consideration here. For example, if the first of the things
you mention can be done without making the code ugly, then I think
that would be worth doing; it's likely to help fairly often in
real-world cases. The problem with making the code contorted and
ugly, as you say that the second idea would require, is that it can
easily mask bugs.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 13:30 Alvaro Herrera <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-21 13:30 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Amit Kapila wrote:
> I think it is because heap_getattr() is not that cheap. We have
> noticed the similar problem during development of scan key push down
> work [1].
One possibility to reduce the cost of that is to use whole tuple deform
instead of repeated individual heap_getattr() calls. Since we don't
actually need *all* attrs, we can create a version of heap_deform_tuple
that takes an attribute number as argument and decodes up to that point.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 14:01 Amit Kapila <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Amit Kapila @ 2017-03-21 14:01 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee
> <[email protected]> wrote:
>>> Yeah. So what's the deal with this? Is somebody working on figuring
>>> out a different approach that would reduce this overhead? Are we
>>> going to defer WARM to v11? Or is the intent to just ignore the 5-10%
>>> slowdown on a single column update and commit everything anyway?
>>
>> I think I should clarify something. The test case does a single column
>> update, but it also has columns which are very wide, has an index on many
>> columns (and it updates a column early in the list). In addition, in the
>> test Mithun updated all 10million rows of the table in a single transaction,
>> used UNLOGGED table and fsync was turned off.
>>
>> TBH I see many artificial scenarios here. It will be very useful if he can
>> rerun the query with some of these restrictions lifted. I'm all for
>> addressing whatever we can, but I am not sure if this test demonstrates a
>> real world usage.
>
> That's a very fair point, but if these patches - or some of them - are
> going to get committed then these things need to get discussed. Let's
> not just have nothing-nothing-nothing giant unagreed code drop.
>
> I think that very wide columns and highly indexed tables are not
> particularly unrealistic, nor do I think updating all the rows is
> particularly unrealistic. Sure, it's not everything, but it's
> something. Now, I would agree that all of that PLUS unlogged tables
> with fsync=off is not too realistic. What kind of regression would we
> observe if we eliminated those last two variables?
>
Sure, we can try that. I think we need to try it with
synchronous_commit = off, otherwise, WAL writes completely overshadows
everything.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 14:12 Robert Haas <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Robert Haas @ 2017-03-21 14:12 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila <[email protected]> wrote:
>> I think that very wide columns and highly indexed tables are not
>> particularly unrealistic, nor do I think updating all the rows is
>> particularly unrealistic. Sure, it's not everything, but it's
>> something. Now, I would agree that all of that PLUS unlogged tables
>> with fsync=off is not too realistic. What kind of regression would we
>> observe if we eliminated those last two variables?
>
> Sure, we can try that. I think we need to try it with
> synchronous_commit = off, otherwise, WAL writes completely overshadows
> everything.
synchronous_commit = off is a much more realistic scenario than fsync = off.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 14:19 Pavan Deolasee <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-21 14:19 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas <[email protected]> wrote:
>
> I think that very wide columns and highly indexed tables are not
> particularly unrealistic, nor do I think updating all the rows is
> particularly unrealistic.
Ok. But those who update 10M rows in a single transaction, would they
really notice 5-10% variation? I think it probably makes sense to run those
updates in smaller transactions and see if the regression is still visible
(otherwise tweaking synchronous_commit is mute anyways).
> Sure, it's not everything, but it's
> something. Now, I would agree that all of that PLUS unlogged tables
> with fsync=off is not too realistic. What kind of regression would we
> observe if we eliminated those last two variables?
>
Hard to say. I didn't find any regression on the machines available to me
even with the original test case that I used, which was pretty bad case to
start with (sure, Mithun tweaked it further to create even worse scenario).
May be the kind of machines he has access to, it might show up even with
those changes.
>
>
> I think that whether the code ends up getting contorted is an
> important consideration here. For example, if the first of the things
> you mention can be done without making the code ugly, then I think
> that would be worth doing; it's likely to help fairly often in
> real-world cases. The problem with making the code contorted and
> ugly, as you say that the second idea would require, is that it can
> easily mask bugs.
Agree. That's probably one reason why Alvaro wrote the patch to start with.
I'll give the first of those two options a try.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 14:21 Alvaro Herrera <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-21 14:21 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Amit Kapila <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Robert Haas wrote:
> On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila <[email protected]> wrote:
> > Sure, we can try that. I think we need to try it with
> > synchronous_commit = off, otherwise, WAL writes completely overshadows
> > everything.
>
> synchronous_commit = off is a much more realistic scenario than fsync = off.
Sure, synchronous_commit=off is a reasonable case. But I say if we lose
a few % on the case where you update only the first indexed of a large
number of very wide columns all indexed, and this is only noticeable if
you don't write WAL and only if you update all the rows in the table,
then I don't see much reason for concern.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 14:40 Robert Haas <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Robert Haas @ 2017-03-21 14:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 10:21 AM, Alvaro Herrera
<[email protected]> wrote:
> Robert Haas wrote:
>> On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila <[email protected]> wrote:
>
>> > Sure, we can try that. I think we need to try it with
>> > synchronous_commit = off, otherwise, WAL writes completely overshadows
>> > everything.
>>
>> synchronous_commit = off is a much more realistic scenario than fsync = off.
>
> Sure, synchronous_commit=off is a reasonable case. But I say if we lose
> a few % on the case where you update only the first indexed of a large
> number of very wide columns all indexed, and this is only noticeable if
> you don't write WAL and only if you update all the rows in the table,
> then I don't see much reason for concern.
If the WAL writing hides the loss, then I agree that's not a big
concern. But if the loss is still visible even when WAL is written,
then I'm not so sure.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 16:10 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Andres Freund @ 2017-03-21 16:10 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Pavan Deolasee <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On 2017-03-21 08:04:11 -0400, Robert Haas wrote:
> On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila <[email protected]> wrote:
> >> Hmm, that test case isn't all that synthetic. It's just a single
> >> column bulk update, which isn't anything all that crazy, and 5-10%
> >> isn't nothing.
> >>
> >> I'm kinda surprised it made that much difference, though.
> >>
> >
> > I think it is because heap_getattr() is not that cheap. We have
> > noticed the similar problem during development of scan key push down
> > work [1].
>
> Yeah. So what's the deal with this? Is somebody working on figuring
> out a different approach that would reduce this overhead?
I think one reasonable thing would be to use slots here, and use
slot_getsomeattrs(), with a pre-computed offset, for doing the
deforming. Given that more than one place run into the issue with
deforming cost via heap_*, that seems like something we're going to have
to do. Additionally the patches I had for JITed deforming all
integrated at the slot layer, so it'd be a good thing from that angle as
well.
Deforming all columns at once would also a boon for the accompanying
index_getattr calls.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 16:21 Andres Freund <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Andres Freund @ 2017-03-21 16:21 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On 2017-03-21 19:49:07 +0530, Pavan Deolasee wrote:
> On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas <[email protected]> wrote:
>
> >
> > I think that very wide columns and highly indexed tables are not
> > particularly unrealistic, nor do I think updating all the rows is
> > particularly unrealistic.
>
>
> Ok. But those who update 10M rows in a single transaction, would they
> really notice 5-10% variation?
Yes. It's very common in ETL, and that's quite performance sensitive.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 16:49 Bruce Momjian <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 16:49 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote:
> On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee
> > TBH I see many artificial scenarios here. It will be very useful if he can
> > rerun the query with some of these restrictions lifted. I'm all for
> > addressing whatever we can, but I am not sure if this test demonstrates a
> > real world usage.
>
> That's a very fair point, but if these patches - or some of them - are
> going to get committed then these things need to get discussed. Let's
> not just have nothing-nothing-nothing giant unagreed code drop.
First, let me say I love this feature for PG 10, along with
multi-variate statistics.
However, not to be a bummer on this, but the persistent question I have
is whether we are locking ourselves into a feature that can only do
_one_ index-change per WARM chain before a lazy vacuum is required. Are
we ever going to figure out how to do more changes per WARM chain in the
future, and is our use of so many bits for this feature going to
restrict our ability to do that in the future.
I know we have talked about it, but not recently, and if everyone else
is fine with it, I am too, but I have to ask these questions.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 17:04 Robert Haas <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 3 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-21 17:04 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 12:49 PM, Bruce Momjian <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote:
>> On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee
>> > TBH I see many artificial scenarios here. It will be very useful if he can
>> > rerun the query with some of these restrictions lifted. I'm all for
>> > addressing whatever we can, but I am not sure if this test demonstrates a
>> > real world usage.
>>
>> That's a very fair point, but if these patches - or some of them - are
>> going to get committed then these things need to get discussed. Let's
>> not just have nothing-nothing-nothing giant unagreed code drop.
>
> First, let me say I love this feature for PG 10, along with
> multi-variate statistics.
>
> However, not to be a bummer on this, but the persistent question I have
> is whether we are locking ourselves into a feature that can only do
> _one_ index-change per WARM chain before a lazy vacuum is required. Are
> we ever going to figure out how to do more changes per WARM chain in the
> future, and is our use of so many bits for this feature going to
> restrict our ability to do that in the future.
>
> I know we have talked about it, but not recently, and if everyone else
> is fine with it, I am too, but I have to ask these questions.
I think that's a good question. I previously expressed similar
concerns. On the one hand, it's hard to ignore the fact that, in the
cases where this wins, it already buys us a lot of performance
improvement. On the other hand, as you say (and as I said), it eats
up a lot of bits, and that limits what we can do in the future. On
the one hand, there is a saying that a bird in the hand is worth two
in the bush. On the other hand, there is also a saying that one
should not paint oneself into the corner.
I'm not sure we've had any really substantive discussion of these
issues. Pavan's response to my previous comments was basically "well,
I think it's worth it", which is entirely reasonable, because he
presumably wouldn't have written the patch that way if he thought it
sucked. But it might not be the only opinion.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 17:08 Peter Geoghegan <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Peter Geoghegan @ 2017-03-21 17:08 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas <[email protected]> wrote:
> I think that's a good question. I previously expressed similar
> concerns. On the one hand, it's hard to ignore the fact that, in the
> cases where this wins, it already buys us a lot of performance
> improvement. On the other hand, as you say (and as I said), it eats
> up a lot of bits, and that limits what we can do in the future. On
> the one hand, there is a saying that a bird in the hand is worth two
> in the bush. On the other hand, there is also a saying that one
> should not paint oneself into the corner.
Are we really saying that there can be no incompatible change to the
on-disk representation for the rest of eternity? I can see why that's
something to avoid indefinitely, but I wouldn't like to rule it out.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 17:14 Robert Haas <[email protected]>
parent: Peter Geoghegan <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-21 17:14 UTC (permalink / raw)
To: Peter Geoghegan <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas <[email protected]> wrote:
>> I think that's a good question. I previously expressed similar
>> concerns. On the one hand, it's hard to ignore the fact that, in the
>> cases where this wins, it already buys us a lot of performance
>> improvement. On the other hand, as you say (and as I said), it eats
>> up a lot of bits, and that limits what we can do in the future. On
>> the one hand, there is a saying that a bird in the hand is worth two
>> in the bush. On the other hand, there is also a saying that one
>> should not paint oneself into the corner.
>
> Are we really saying that there can be no incompatible change to the
> on-disk representation for the rest of eternity? I can see why that's
> something to avoid indefinitely, but I wouldn't like to rule it out.
Well, I don't want to rule it out either, but if we do a release to
which you can't pg_upgrade, it's going to be really painful for a lot
of users. Many users can't realistically upgrade using pg_dump, ever.
So they'll be stuck on the release before the one that breaks
compatibility for a very long time.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 17:17 Bruce Momjian <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 17:17 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 01:04:14PM -0400, Robert Haas wrote:
> > I know we have talked about it, but not recently, and if everyone else
> > is fine with it, I am too, but I have to ask these questions.
>
> I think that's a good question. I previously expressed similar
> concerns. On the one hand, it's hard to ignore the fact that, in the
> cases where this wins, it already buys us a lot of performance
> improvement. On the other hand, as you say (and as I said), it eats
> up a lot of bits, and that limits what we can do in the future. On
> the one hand, there is a saying that a bird in the hand is worth two
> in the bush. On the other hand, there is also a saying that one
> should not paint oneself into the corner.
>
> I'm not sure we've had any really substantive discussion of these
> issues. Pavan's response to my previous comments was basically "well,
> I think it's worth it", which is entirely reasonable, because he
> presumably wouldn't have written the patch that way if he thought it
> sucked. But it might not be the only opinion.
Early in the discussion we talked about allowing multiple changes per
WARM chain if they all changed the same index and were in the same
direction so there were no duplicates, but it was complicated. There
was also discussion about checking the index during INSERT/UPDATE to see
if there was a duplicate. However, those ideas never led to further
discussion.
I know the current patch yields good results, but only on a narrow test
case, so I am not ready to just stop asking questions based the opinion
of the author or test results alone.
If someone came to me and said, "We have thought about allowing more
than one index change per WARM chain, and if we can ever do it, it will
probably be done this way, and we have the bits for it," I would be more
comfortable.
One interesting side-issue is that indirect indexes have a similar
problem with duplicate index entries, and there is no plan on how to fix
that either. I guess I just don't feel we have explored the
duplicate-index-entry problem enough for me to be comfortable.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 17:19 Bruce Momjian <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 17:19 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 01:14:00PM -0400, Robert Haas wrote:
> On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan <[email protected]> wrote:
> > On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas <[email protected]> wrote:
> >> I think that's a good question. I previously expressed similar
> >> concerns. On the one hand, it's hard to ignore the fact that, in the
> >> cases where this wins, it already buys us a lot of performance
> >> improvement. On the other hand, as you say (and as I said), it eats
> >> up a lot of bits, and that limits what we can do in the future. On
> >> the one hand, there is a saying that a bird in the hand is worth two
> >> in the bush. On the other hand, there is also a saying that one
> >> should not paint oneself into the corner.
> >
> > Are we really saying that there can be no incompatible change to the
> > on-disk representation for the rest of eternity? I can see why that's
> > something to avoid indefinitely, but I wouldn't like to rule it out.
>
> Well, I don't want to rule it out either, but if we do a release to
> which you can't pg_upgrade, it's going to be really painful for a lot
> of users. Many users can't realistically upgrade using pg_dump, ever.
> So they'll be stuck on the release before the one that breaks
> compatibility for a very long time.
Right. If we weren't setting tuple and tid bits we could imrpove it
easily in PG 11, but if we use them for a single-change WARM chain for
PG 10, we might need bits that are not available to improve it later.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:03 Petr Jelinek <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Petr Jelinek @ 2017-03-21 18:03 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On 21/03/17 18:14, Robert Haas wrote:
> On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan <[email protected]> wrote:
>> On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas <[email protected]> wrote:
>>> I think that's a good question. I previously expressed similar
>>> concerns. On the one hand, it's hard to ignore the fact that, in the
>>> cases where this wins, it already buys us a lot of performance
>>> improvement. On the other hand, as you say (and as I said), it eats
>>> up a lot of bits, and that limits what we can do in the future. On
>>> the one hand, there is a saying that a bird in the hand is worth two
>>> in the bush. On the other hand, there is also a saying that one
>>> should not paint oneself into the corner.
>>
>> Are we really saying that there can be no incompatible change to the
>> on-disk representation for the rest of eternity? I can see why that's
>> something to avoid indefinitely, but I wouldn't like to rule it out.
>
> Well, I don't want to rule it out either, but if we do a release to
> which you can't pg_upgrade, it's going to be really painful for a lot
> of users. Many users can't realistically upgrade using pg_dump, ever.
> So they'll be stuck on the release before the one that breaks
> compatibility for a very long time.
>
This is why I like the idea of pluggable storage, if we ever get that it
would buy us ability to implement completely different heap format
without breaking pg_upgrade.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:05 Petr Jelinek <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Petr Jelinek @ 2017-03-21 18:05 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; Robert Haas <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On 21/03/17 18:19, Bruce Momjian wrote:
> On Tue, Mar 21, 2017 at 01:14:00PM -0400, Robert Haas wrote:
>> On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan <[email protected]> wrote:
>>> On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas <[email protected]> wrote:
>>>> I think that's a good question. I previously expressed similar
>>>> concerns. On the one hand, it's hard to ignore the fact that, in the
>>>> cases where this wins, it already buys us a lot of performance
>>>> improvement. On the other hand, as you say (and as I said), it eats
>>>> up a lot of bits, and that limits what we can do in the future. On
>>>> the one hand, there is a saying that a bird in the hand is worth two
>>>> in the bush. On the other hand, there is also a saying that one
>>>> should not paint oneself into the corner.
>>>
>>> Are we really saying that there can be no incompatible change to the
>>> on-disk representation for the rest of eternity? I can see why that's
>>> something to avoid indefinitely, but I wouldn't like to rule it out.
>>
>> Well, I don't want to rule it out either, but if we do a release to
>> which you can't pg_upgrade, it's going to be really painful for a lot
>> of users. Many users can't realistically upgrade using pg_dump, ever.
>> So they'll be stuck on the release before the one that breaks
>> compatibility for a very long time.
>
> Right. If we weren't setting tuple and tid bits we could imrpove it
> easily in PG 11, but if we use them for a single-change WARM chain for
> PG 10, we might need bits that are not available to improve it later.
>
I thought there is still couple of bits available.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:15 Pavan Deolasee <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-21 18:15 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 10:47 PM, Bruce Momjian <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 01:04:14PM -0400, Robert Haas wrote:
> > > I know we have talked about it, but not recently, and if everyone else
> > > is fine with it, I am too, but I have to ask these questions.
> >
> > I think that's a good question. I previously expressed similar
> > concerns. On the one hand, it's hard to ignore the fact that, in the
> > cases where this wins, it already buys us a lot of performance
> > improvement. On the other hand, as you say (and as I said), it eats
> > up a lot of bits, and that limits what we can do in the future. On
> > the one hand, there is a saying that a bird in the hand is worth two
> > in the bush. On the other hand, there is also a saying that one
> > should not paint oneself into the corner.
> >
> > I'm not sure we've had any really substantive discussion of these
> > issues. Pavan's response to my previous comments was basically "well,
> > I think it's worth it", which is entirely reasonable, because he
> > presumably wouldn't have written the patch that way if he thought it
> > sucked. But it might not be the only opinion.
>
> Early in the discussion we talked about allowing multiple changes per
> WARM chain if they all changed the same index and were in the same
> direction so there were no duplicates, but it was complicated. There
> was also discussion about checking the index during INSERT/UPDATE to see
> if there was a duplicate. However, those ideas never led to further
> discussion.
>
Well, once I started thinking about how to do vacuum etc, I realised that
any mechanism which allows unlimited (even handful) updates per chain is
going to be very complex and error prone. But if someone has ideas to do
that, I am open. I must say though, it will make an already complex problem
even more complex.
>
> I know the current patch yields good results, but only on a narrow test
> case,
Hmm. I am kinda surprised you say that because I never thought it was a
narrow test case that we are targeting here. But may be I'm wrong.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:17 Robert Haas <[email protected]>
parent: Petr Jelinek <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Robert Haas @ 2017-03-21 18:17 UTC (permalink / raw)
To: Petr Jelinek <[email protected]>; +Cc: Peter Geoghegan <[email protected]>; Bruce Momjian <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 2:03 PM, Petr Jelinek
<[email protected]> wrote:
> This is why I like the idea of pluggable storage, if we ever get that it
> would buy us ability to implement completely different heap format
> without breaking pg_upgrade.
You probably won't be surprised to hear that I agree. :-)
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:24 Pavan Deolasee <[email protected]>
parent: Robert Haas <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-21 18:24 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 10:34 PM, Robert Haas <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 12:49 PM, Bruce Momjian <[email protected]> wrote:
> > On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote:
> >> On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee
> >> > TBH I see many artificial scenarios here. It will be very useful if
> he can
> >> > rerun the query with some of these restrictions lifted. I'm all for
> >> > addressing whatever we can, but I am not sure if this test
> demonstrates a
> >> > real world usage.
> >>
> >> That's a very fair point, but if these patches - or some of them - are
> >> going to get committed then these things need to get discussed. Let's
> >> not just have nothing-nothing-nothing giant unagreed code drop.
> >
> > First, let me say I love this feature for PG 10, along with
> > multi-variate statistics.
> >
> > However, not to be a bummer on this, but the persistent question I have
> > is whether we are locking ourselves into a feature that can only do
> > _one_ index-change per WARM chain before a lazy vacuum is required. Are
> > we ever going to figure out how to do more changes per WARM chain in the
> > future, and is our use of so many bits for this feature going to
> > restrict our ability to do that in the future.
> >
> > I know we have talked about it, but not recently, and if everyone else
> > is fine with it, I am too, but I have to ask these questions.
>
> I think that's a good question. I previously expressed similar
> concerns. On the one hand, it's hard to ignore the fact that, in the
> cases where this wins, it already buys us a lot of performance
> improvement. On the other hand, as you say (and as I said), it eats
> up a lot of bits, and that limits what we can do in the future.
I think we can save a bit few bits, at some additional costs and/or
complexity. It all depends on what matters us more. For example, we can
choose not to use HEAP_LATEST_TUPLE bit and instead always find the root
tuple the hard way. Since only WARM would ever need to find that
information, may be it's ok since WARM's other advantage will justify that.
Or we cache the information computed during earlier heap_prune_page call
and use that (just guessing that we can make it work, no concrete idea at
this moment).
We can also save HEAP_WARM_UPDATED flag since this is required only for
abort-handling case. We can find a way to push that information down to the
old tuple if UPDATE aborts and we detect the broken chain. Again, not fully
thought through, but doable. Of course, we will have to carefully evaluate
all code paths and make sure that we don't lose that information ever.
If the consumption of bits become a deal breaker then I would first trade
the HEAP_LATEST_TUPLE bit and then HEAP_WARM_UPDATED just from correctness
perspective.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 18:56 Bruce Momjian <[email protected]>
parent: Petr Jelinek <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 18:56 UTC (permalink / raw)
To: Petr Jelinek <[email protected]>; +Cc: Robert Haas <[email protected]>; Peter Geoghegan <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 07:05:15PM +0100, Petr Jelinek wrote:
> >> Well, I don't want to rule it out either, but if we do a release to
> >> which you can't pg_upgrade, it's going to be really painful for a lot
> >> of users. Many users can't realistically upgrade using pg_dump, ever.
> >> So they'll be stuck on the release before the one that breaks
> >> compatibility for a very long time.
> >
> > Right. If we weren't setting tuple and tid bits we could improve it
> > easily in PG 11, but if we use them for a single-change WARM chain for
> > PG 10, we might need bits that are not available to improve it later.
> >
>
> I thought there is still couple of bits available.
Yes, there are. The issue is that we don't know how we would improve it
so we don't know how many bits we need, and my concern is that we
haven't discussed the improvement ideas enough to know we have done the
best we can for PG 10.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 19:00 Bruce Momjian <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 19:00 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 11:45:09PM +0530, Pavan Deolasee wrote:
> Early in the discussion we talked about allowing multiple changes per
> WARM chain if they all changed the same index and were in the same
> direction so there were no duplicates, but it was complicated. There
> was also discussion about checking the index during INSERT/UPDATE to see
> if there was a duplicate. However, those ideas never led to further
> discussion.
>
>
> Well, once I started thinking about how to do vacuum etc, I realised that any
> mechanism which allows unlimited (even handful) updates per chain is going to
> be very complex and error prone. But if someone has ideas to do that, I am
> open. I must say though, it will make an already complex problem even more
> complex.
Yes, that is where we got stuck. Have enough people studied the issue
to know that there are no simple answers?
> I know the current patch yields good results, but only on a narrow test
> case,
>
>
> Hmm. I am kinda surprised you say that because I never thought it was a narrow
> test case that we are targeting here. But may be I'm wrong.
Well, it is really a question of how often you want to do a second WARM
update (not possible) vs. the frequency of lazy vacuum. I assumed that
would be a 100X or 10kX difference, but I am not sure myself either. My
initial guess was that only allowing a single WARM update between lazy
vacuums would show no improvementin in real-world workloads, but maybe I
am wrong.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 19:04 Bruce Momjian <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 19:04 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 11:54:25PM +0530, Pavan Deolasee wrote:
> We can also save HEAP_WARM_UPDATED flag since this is required only for
> abort-handling case. We can find a way to push that information down to the old
> tuple if UPDATE aborts and we detect the broken chain. Again, not fully thought
> through, but doable. Of course, we will have to carefully evaluate all code
> paths and make sure that we don't lose that information ever.
>
> If the consumption of bits become a deal breaker then I would first trade the
> HEAP_LATEST_TUPLE bit and then HEAP_WARM_UPDATED just from correctness
> perspective.
I don't think it makes sense to try and save bits and add complexity
when we have no idea if we will ever use them, but again, I am back to
my original question of whether we have done sufficient research, and if
everyone says "yes", I am find with that.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 19:43 Alvaro Herrera <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-21 19:43 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Bruce Momjian wrote:
> I don't think it makes sense to try and save bits and add complexity
> when we have no idea if we will ever use them,
If we find ourselves in dire need of additional bits, there is a known
mechanism to get back 2 bits from old-style VACUUM FULL. I assume that
the reason nobody has bothered to write the code for that is that
there's no *that* much interest.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 19:48 Bruce Momjian <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 19:48 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
>
> > I don't think it makes sense to try and save bits and add complexity
> > when we have no idea if we will ever use them,
>
> If we find ourselves in dire need of additional bits, there is a known
> mechanism to get back 2 bits from old-style VACUUM FULL. I assume that
> the reason nobody has bothered to write the code for that is that
> there's no *that* much interest.
We have no way of tracking if users still have pages that used the bits
via pg_upgrade before they were removed.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 19:56 Alvaro Herrera <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Alvaro Herrera @ 2017-03-21 19:56 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Bruce Momjian wrote:
> On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote:
> > Bruce Momjian wrote:
> >
> > > I don't think it makes sense to try and save bits and add complexity
> > > when we have no idea if we will ever use them,
> >
> > If we find ourselves in dire need of additional bits, there is a known
> > mechanism to get back 2 bits from old-style VACUUM FULL. I assume that
> > the reason nobody has bothered to write the code for that is that
> > there's no *that* much interest.
>
> We have no way of tracking if users still have pages that used the bits
> via pg_upgrade before they were removed.
Yes, that's exactly the code that needs to be written.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 20:04 Bruce Momjian <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Bruce Momjian @ 2017-03-21 20:04 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 04:56:16PM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote:
> > > Bruce Momjian wrote:
> > >
> > > > I don't think it makes sense to try and save bits and add complexity
> > > > when we have no idea if we will ever use them,
> > >
> > > If we find ourselves in dire need of additional bits, there is a known
> > > mechanism to get back 2 bits from old-style VACUUM FULL. I assume that
> > > the reason nobody has bothered to write the code for that is that
> > > there's no *that* much interest.
> >
> > We have no way of tracking if users still have pages that used the bits
> > via pg_upgrade before they were removed.
>
> Yes, that's exactly the code that needs to be written.
Yes, but once it is written it will take years before those bits can be
used on most installations.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-21 22:21 Mithun Cy <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Mithun Cy @ 2017-03-21 22:21 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Pavan Deolasee <[email protected]>; Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 8:10 PM, Robert Haas <[email protected]> wrote:
> If the WAL writing hides the loss, then I agree that's not a big
> concern. But if the loss is still visible even when WAL is written,
> then I'm not so sure.
The tests table schema was taken from earlier tests what Pavan has posted
[1], hence it is UNLOGGED all I tried to stress the tests. Instead of
updating 1 row at a time through pgbench (For which I and Pavan both did
not see any regression), I tried to update all the rows in the single
statement. I have changed the settings as recommended and did a quick test
as above in our machine by removing UNLOGGED world in create table
statement.
Patch Tested : Only 0001_interesting_attrs_v18.patch in [2]
Machine: Scylla [ Last time I did same tests on IBM power2 but It is not
immediately available. So trying on another intel based performance
machine.]
============
[mithun.cy@scylla bin]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz
Stepping: 2
CPU MHz: 1235.800
BogoMIPS: 4594.35
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-13,28-41
NUMA node1 CPU(s): 14-27,42-55
[mithun.cy@scylla bin]$ cat /proc/meminfo
MemTotal: 65687464 kB
Postgresql.conf non default settings
===========================
shared_buffers = 24 GB
max_wal_size = 10GB
min_wal_size = 5GB
synchronous_commit=off
autovacuum = off /*manually doing vacumm full before every update. */
This system has 2 storage I have kept datadir on spinning disc and pg_wal
on ssd.
Tests :
DROP TABLE IF EXISTS testtab;
CREATE TABLE testtab (
col1 integer,
col2 text,
col3 float,
col4 text,
col5 text,
col6 char(30),
col7 text,
col8 date,
col9 text,
col10 text
);
INSERT INTO testtab
SELECT generate_series(1,10000000),
md5(random()::text),
random(),
md5(random()::text),
md5(random()::text),
md5(random()::text)::char(30),
md5(random()::text),
now(),
md5(random()::text),
md5(random()::text);
CREATE INDEX testindx ON testtab (col1, col2, col3, col4, col5, col6, col7,
col8, col9);
Performance measurement tests: Ran12 times to eliminate run to run
latencies.
==========================
VACUUM FULL;
BEGIN;
UPDATE testtab SET col2 = md5(random()::text);
ROLLBACK;
Response time recorded shows there is a much higher increase in response
time from 10% to 25% after the patch.
[1] Re: rewrite HeapSatisfiesHOTAndKey
<https://www.postgresql.org/message-id/CABOikdMUQQs4BnJ4Ws-ObOEDh8vhNp13Y1caK_i8seSHKPjbhw%40mail.gma...;
[2] Re: Patch: Write Amplification Reduction Method (WARM)
<https://www.postgresql.org/message-id/CABOikdP1yeicUPH0NByjrg2Sv3ZtJXWyFPSqwppid8G3kLVKjw%40mail.gma...;
--
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/vnd.oasis.opendocument.spreadsheet] WARM_test.ods (11.2K, ../../CAD__Ouips_paapmpuWK47gj7zmcoY8rx7Ynbfu2ywMjdWm7Thg@mail.gmail.com/3-WARM_test.ods)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 03:13 Pavan Deolasee <[email protected]>
parent: Mithun Cy <[email protected]>
1 sibling, 2 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-22 03:13 UTC (permalink / raw)
To: Mithun Cy <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 3:51 AM, Mithun Cy <[email protected]>
wrote:
> On Tue, Mar 21, 2017 at 8:10 PM, Robert Haas <[email protected]>
> wrote:
> > If the WAL writing hides the loss, then I agree that's not a big
> > concern. But if the loss is still visible even when WAL is written,
> > then I'm not so sure.
>
> The tests table schema was taken from earlier tests what Pavan has posted
> [1], hence it is UNLOGGED all I tried to stress the tests. Instead of
> updating 1 row at a time through pgbench (For which I and Pavan both did
> not see any regression), I tried to update all the rows in the single
> statement.
>
Sorry, I did not mean to suggest that you set it up wrongly, I was just
trying to point out that the test case itself may not be very practical.
But given your recent numbers, the regression is clearly non-trivial and
something we must address.
> I have changed the settings as recommended and did a quick test as above
> in our machine by removing UNLOGGED world in create table statement.
>
> Patch Tested : Only 0001_interesting_attrs_v18.patch in [2]
>
> Response time recorded shows there is a much higher increase in response
> time from 10% to 25% after the patch.
>
>
Thanks for repeating the tests. They are very useful. It might make sense
to reverse the order or do 6 tests each and alternate between patched and
unpatched master just to get rid of any other anomaly.
BTW may I request another test with the attached patch? In this patch, we
check if the PageIsFull() even before deciding which attributes to check
for modification. If the page is already full, there is hardly any chance
of doing a HOT update (there could be a corner case where the new tuple is
smaller than the tuple used in previous UPDATE and we have just enough
space to do HOT update this time, but I can think that's too narrow).
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0001_interesting_attrs_v19.patch (12.5K, ../../CABOikdN6r2uoVB6QRrBz4OYsGgqNqZcmyQOTC=WSzSRuWmTaaw@mail.gmail.com/3-0001_interesting_attrs_v19.patch)
download | inline diff:
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 8526137..36f7ac8 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -96,11 +96,8 @@ static XLogRecPtr log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup,
HeapTuple newtup, HeapTuple old_key_tup,
bool all_visible_cleared, bool new_all_visible_cleared);
-static void HeapSatisfiesHOTandKeyUpdate(Relation relation,
- Bitmapset *hot_attrs,
- Bitmapset *key_attrs, Bitmapset *id_attrs,
- bool *satisfies_hot, bool *satisfies_key,
- bool *satisfies_id,
+static Bitmapset *HeapDetermineModifiedColumns(Relation relation,
+ Bitmapset *interesting_cols,
HeapTuple oldtup, HeapTuple newtup);
static bool heap_acquire_tuplock(Relation relation, ItemPointer tid,
LockTupleMode mode, LockWaitPolicy wait_policy,
@@ -3471,6 +3468,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Bitmapset *hot_attrs;
Bitmapset *key_attrs;
Bitmapset *id_attrs;
+ Bitmapset *interesting_attrs;
+ Bitmapset *modified_attrs;
ItemId lp;
HeapTupleData oldtup;
HeapTuple heaptup;
@@ -3488,10 +3487,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
pagefree;
bool have_tuple_lock = false;
bool iscombo;
- bool satisfies_hot;
- bool satisfies_key;
- bool satisfies_id;
bool use_hot_update = false;
+ bool hot_attrs_checked = false;
bool key_intact;
bool all_visible_cleared = false;
bool all_visible_cleared_new = false;
@@ -3517,26 +3514,51 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
errmsg("cannot update tuples during a parallel operation")));
/*
- * Fetch the list of attributes to be checked for HOT update. This is
- * wasted effort if we fail to update or have to put the new tuple on a
- * different page. But we must compute the list before obtaining buffer
- * lock --- in the worst case, if we are doing an update on one of the
- * relevant system catalogs, we could deadlock if we try to fetch the list
- * later. In any case, the relcache caches the data so this is usually
- * pretty cheap.
+ * Fetch the list of attributes to be checked for various operations.
*
- * Note that we get a copy here, so we need not worry about relcache flush
- * happening midway through.
+ * For HOT considerations, this is wasted effort if we fail to update or
+ * have to put the new tuple on a different page. But we must compute the
+ * list before obtaining buffer lock --- in the worst case, if we are doing
+ * an update on one of the relevant system catalogs, we could deadlock if
+ * we try to fetch the list later. In any case, the relcache caches the
+ * data so this is usually pretty cheap.
+ *
+ * We also need columns used by the replica identity, the columns that
+ * are considered the "key" of rows in the table, and columns that are
+ * part of indirect indexes.
+ *
+ * Note that we get copies of each bitmap, so we need not worry about
+ * relcache flush happening midway through.
*/
hot_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_ALL);
key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);
id_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
block = ItemPointerGetBlockNumber(otid);
buffer = ReadBuffer(relation, block);
page = BufferGetPage(buffer);
+ interesting_attrs = NULL;
+ /*
+ * If the page is already full, there is hardly any chance of doing a HOT
+ * update on this page. It might be wasteful effort to look for index
+ * column updates only to later reject HOT updates for lack of space in the
+ * same page. So we be conservative and only fetch hot_attrs if the page is
+ * not already full. Since we are already holding a pin on the buffer,
+ * there is no chance that the buffer can get cleaned up concurrently and
+ * even if that was possible, in the worst case we lose a chance to do a
+ * HOT update.
+ */
+ if (!PageIsFull(page))
+ {
+ interesting_attrs = bms_add_members(interesting_attrs, hot_attrs);
+ hot_attrs_checked = true;
+ }
+ interesting_attrs = bms_add_members(interesting_attrs, key_attrs);
+ interesting_attrs = bms_add_members(interesting_attrs, id_attrs);
+
/*
* Before locking the buffer, pin the visibility map page if it appears to
* be necessary. Since we haven't got the lock yet, someone else might be
@@ -3552,7 +3574,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Assert(ItemIdIsNormal(lp));
/*
- * Fill in enough data in oldtup for HeapSatisfiesHOTandKeyUpdate to work
+ * Fill in enough data in oldtup for HeapDetermineModifiedColumns to work
* properly.
*/
oldtup.t_tableOid = RelationGetRelid(relation);
@@ -3578,6 +3600,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
Assert(!(newtup->t_data->t_infomask & HEAP_HASOID));
}
+ /* Determine columns modified by the update. */
+ modified_attrs = HeapDetermineModifiedColumns(relation, interesting_attrs,
+ &oldtup, newtup);
+
/*
* If we're not updating any "key" column, we can grab a weaker lock type.
* This allows for more concurrency when we are running simultaneously
@@ -3589,10 +3615,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
* is updates that don't manipulate key columns, not those that
* serendipitiously arrive at the same key values.
*/
- HeapSatisfiesHOTandKeyUpdate(relation, hot_attrs, key_attrs, id_attrs,
- &satisfies_hot, &satisfies_key,
- &satisfies_id, &oldtup, newtup);
- if (satisfies_key)
+ if (!bms_overlap(modified_attrs, key_attrs))
{
*lockmode = LockTupleNoKeyExclusive;
mxact_status = MultiXactStatusNoKeyUpdate;
@@ -3831,6 +3854,8 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(modified_attrs);
+ bms_free(interesting_attrs);
return result;
}
@@ -4133,9 +4158,10 @@ l2:
/*
* Since the new tuple is going into the same page, we might be able
* to do a HOT update. Check if any of the index columns have been
- * changed. If not, then HOT update is possible.
+ * changed. If the page was already full, we may have skipped checking
+ * for index columns. If so, HOT update is possible.
*/
- if (satisfies_hot)
+ if (hot_attrs_checked && !bms_overlap(modified_attrs, hot_attrs))
use_hot_update = true;
}
else
@@ -4150,7 +4176,9 @@ l2:
* ExtractReplicaIdentity() will return NULL if nothing needs to be
* logged.
*/
- old_key_tuple = ExtractReplicaIdentity(relation, &oldtup, !satisfies_id, &old_key_copied);
+ old_key_tuple = ExtractReplicaIdentity(relation, &oldtup,
+ bms_overlap(modified_attrs, id_attrs),
+ &old_key_copied);
/* NO EREPORT(ERROR) from here till changes are logged */
START_CRIT_SECTION();
@@ -4298,13 +4326,15 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(modified_attrs);
+ bms_free(interesting_attrs);
return HeapTupleMayBeUpdated;
}
/*
* Check if the specified attribute's value is same in both given tuples.
- * Subroutine for HeapSatisfiesHOTandKeyUpdate.
+ * Subroutine for HeapDetermineModifiedColumns.
*/
static bool
heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
@@ -4338,7 +4368,7 @@ heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
/*
* Extract the corresponding values. XXX this is pretty inefficient if
- * there are many indexed columns. Should HeapSatisfiesHOTandKeyUpdate do
+ * there are many indexed columns. Should HeapDetermineModifiedColumns do
* a single heap_deform_tuple call on each tuple, instead? But that
* doesn't work for system columns ...
*/
@@ -4383,114 +4413,30 @@ heap_tuple_attr_equals(TupleDesc tupdesc, int attrnum,
/*
* Check which columns are being updated.
*
- * This simultaneously checks conditions for HOT updates, for FOR KEY
- * SHARE updates, and REPLICA IDENTITY concerns. Since much of the time they
- * will be checking very similar sets of columns, and doing the same tests on
- * them, it makes sense to optimize and do them together.
- *
- * We receive three bitmapsets comprising the three sets of columns we're
- * interested in. Note these are destructively modified; that is OK since
- * this is invoked at most once in heap_update.
+ * Given an updated tuple, determine (and return into the output bitmapset),
+ * from those listed as interesting, the set of columns that changed.
*
- * hot_result is set to TRUE if it's okay to do a HOT update (i.e. it does not
- * modified indexed columns); key_result is set to TRUE if the update does not
- * modify columns used in the key; id_result is set to TRUE if the update does
- * not modify columns in any index marked as the REPLICA IDENTITY.
+ * The input bitmapset is destructively modified; that is OK since this is
+ * invoked at most once in heap_update.
*/
-static void
-HeapSatisfiesHOTandKeyUpdate(Relation relation, Bitmapset *hot_attrs,
- Bitmapset *key_attrs, Bitmapset *id_attrs,
- bool *satisfies_hot, bool *satisfies_key,
- bool *satisfies_id,
+static Bitmapset *
+HeapDetermineModifiedColumns(Relation relation, Bitmapset *interesting_cols,
HeapTuple oldtup, HeapTuple newtup)
{
- int next_hot_attnum;
- int next_key_attnum;
- int next_id_attnum;
- bool hot_result = true;
- bool key_result = true;
- bool id_result = true;
-
- /* If REPLICA IDENTITY is set to FULL, id_attrs will be empty. */
- Assert(bms_is_subset(id_attrs, key_attrs));
- Assert(bms_is_subset(key_attrs, hot_attrs));
-
- /*
- * If one of these sets contains no remaining bits, bms_first_member will
- * return -1, and after adding FirstLowInvalidHeapAttributeNumber (which
- * is negative!) we'll get an attribute number that can't possibly be
- * real, and thus won't match any actual attribute number.
- */
- next_hot_attnum = bms_first_member(hot_attrs);
- next_hot_attnum += FirstLowInvalidHeapAttributeNumber;
- next_key_attnum = bms_first_member(key_attrs);
- next_key_attnum += FirstLowInvalidHeapAttributeNumber;
- next_id_attnum = bms_first_member(id_attrs);
- next_id_attnum += FirstLowInvalidHeapAttributeNumber;
+ int attnum;
+ Bitmapset *modified = NULL;
- for (;;)
+ while ((attnum = bms_first_member(interesting_cols)) >= 0)
{
- bool changed;
- int check_now;
+ attnum += FirstLowInvalidHeapAttributeNumber;
- /*
- * Since the HOT attributes are a superset of the key attributes and
- * the key attributes are a superset of the id attributes, this logic
- * is guaranteed to identify the next column that needs to be checked.
- */
- if (hot_result && next_hot_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_hot_attnum;
- else if (key_result && next_key_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_key_attnum;
- else if (id_result && next_id_attnum > FirstLowInvalidHeapAttributeNumber)
- check_now = next_id_attnum;
- else
- break;
-
- /* See whether it changed. */
- changed = !heap_tuple_attr_equals(RelationGetDescr(relation),
- check_now, oldtup, newtup);
- if (changed)
- {
- if (check_now == next_hot_attnum)
- hot_result = false;
- if (check_now == next_key_attnum)
- key_result = false;
- if (check_now == next_id_attnum)
- id_result = false;
-
- /* if all are false now, we can stop checking */
- if (!hot_result && !key_result && !id_result)
- break;
- }
-
- /*
- * Advance the next attribute numbers for the sets that contain the
- * attribute we just checked. As we work our way through the columns,
- * the next_attnum values will rise; but when each set becomes empty,
- * bms_first_member() will return -1 and the attribute number will end
- * up with a value less than FirstLowInvalidHeapAttributeNumber.
- */
- if (hot_result && check_now == next_hot_attnum)
- {
- next_hot_attnum = bms_first_member(hot_attrs);
- next_hot_attnum += FirstLowInvalidHeapAttributeNumber;
- }
- if (key_result && check_now == next_key_attnum)
- {
- next_key_attnum = bms_first_member(key_attrs);
- next_key_attnum += FirstLowInvalidHeapAttributeNumber;
- }
- if (id_result && check_now == next_id_attnum)
- {
- next_id_attnum = bms_first_member(id_attrs);
- next_id_attnum += FirstLowInvalidHeapAttributeNumber;
- }
+ if (!heap_tuple_attr_equals(RelationGetDescr(relation),
+ attnum, oldtup, newtup))
+ modified = bms_add_member(modified,
+ attnum - FirstLowInvalidHeapAttributeNumber);
}
- *satisfies_hot = hot_result;
- *satisfies_key = key_result;
- *satisfies_id = id_result;
+ return modified;
}
/*
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 07:50 Pavan Deolasee <[email protected]>
parent: Pavan Deolasee <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-22 07:50 UTC (permalink / raw)
To: Mithun Cy <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 8:43 AM, Pavan Deolasee <[email protected]>
wrote:
>
>
> BTW may I request another test with the attached patch? In this patch, we
> check if the PageIsFull() even before deciding which attributes to check
> for modification. If the page is already full, there is hardly any chance
> of doing a HOT update (there could be a corner case where the new tuple is
> smaller than the tuple used in previous UPDATE and we have just enough
> space to do HOT update this time, but I can think that's too narrow).
>
>
I would also request you to do a slightly different test where instead of
updating the second column, we update the last column of the index i.e.
col9. Would really appreciate if you share results with both master and v19
patch.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 10:14 Pavan Deolasee <[email protected]>
parent: Mithun Cy <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-22 10:14 UTC (permalink / raw)
To: Mithun Cy <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 3:51 AM, Mithun Cy <[email protected]>
wrote:
>
> CREATE INDEX testindx ON testtab (col1, col2, col3, col4, col5, col6,
> col7, col8, col9);
> Performance measurement tests: Ran12 times to eliminate run to run
> latencies.
> ==========================
> VACUUM FULL;
> BEGIN;
> UPDATE testtab SET col2 = md5(random()::text);
> ROLLBACK;
>
> Response time recorded shows there is a much higher increase in response
> time from 10% to 25% after the patch.
>
>
After doing some tests on my side, I now think that there is something else
going on, unrelated to the patch. I ran the same benchmark on AWS i2.xlarge
machine with 32GB RAM. shared_buffers set to 16GB, max_wal_size to 256GB,
checkpoint_timeout to 60min and autovacuum off.
I compared master and v19, every time running 6 runs of the test. The
database was restarted whenever changing binaries, tables dropped/recreated
and checkpoint taken after each restart (but not between 2 runs, which I
believe what you did too.. but correct me if that's a wrong assumption).
Instead of col2, I am updating col9, but that's probably not too much
relevant.
VACUUM FULL;
BEGIN;
UPDATE testtab SET col9 = md5(random()::text);
ROLLBACK;
First set of 6 runs with master:
163629.8
181183.8
194788.1
194606.1
194589.9
196002.6
(database restart, table drop/create, checkpoint)
First set of 6 runs with v19:
190566.55
228274.489
238110.202
239304.681
258748.189
284882.4
(database restart, table drop/create, checkpoint)
Second set of 6 runs with master:
232267.5
298259.6
312315.1
341817.3
360729.2
385210.7
This looks quite weird to me. Obviously these numbers are completely
non-comparable. Even the time for VACUUM FULL goes up with every run.
May be we can blame it on AWS instance completely, but the pattern in your
tests looks very similar where the number slowly and steadily keeps going
up. If you do complete retest but run v18/v19 first and then run master,
may be we'll see a complete opposite picture?
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 10:36 Amit Kapila <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 2 replies; 104+ messages in thread
From: Amit Kapila @ 2017-03-22 10:36 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee
<[email protected]> wrote:
>
>>
>
> Please find attached rebased patches.
>
Few comments on 0005_warm_updates_v18.patch:
1.
@@ -806,20 +835,35 @@ hashbucketcleanup(Relation rel, Bucket
cur_bucket, Buffer bucket_buf,
{
..
- if (callback && callback(htup, callback_state))
+ if(callback)
{
- kill_tuple = true;
-
if (tuples_removed)
- *tuples_removed += 1;
+result = callback(htup, is_warm, callback_state);
+ if (result== IBDCR_DELETE)
+ {
+ kill_tuple = true;
+ if (tuples_removed)
+*tuples_removed += 1;
+ }
+ else if (result ==IBDCR_CLEAR_WARM)
+ {
+ clear_tuple= true;
+ }
}
else if
(split_cleanup)
..
}
I think this will break the existing mechanism of split cleanup. We
need to check for split cleanup if the tuple is tuple is not deletable
by the callback. This is not merely an optimization but a must
condition because we will clear the split cleanup flag after this
bucket is scanned completely.
2.
- PageIndexMultiDelete(page, deletable, ndeletable);
+ /*
+
* Clear the WARM pointers.
+ *
+ * We mustdo this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the
+ * array and hence offnums recorded earlierwon't make any sense
+ * after PageIndexMultiDelete is called.
+
*/
+ if (nclearwarm > 0)
+ _hash_clear_items(page,clearwarm, nclearwarm);
+
+ /*
+ * And delete the deletableitems
+ */
+ if (ndeletable > 0)
+
PageIndexMultiDelete(page, deletable, ndeletable);
I think this assumes that the items where we need to clear warm flag
are not deletable, otherwise what is the need to clear the flag if we
are going to delete the tuple. The deletable tuple can have a warm
flag if it is deletable due to split cleanup.
3.
+ /*
+ * HASH indexes compute a hash value of the key and store that in the
+ * index. So
we must first obtain the hash of the value obtained from the
+ * heap and then do a comparison
+
*/
+ _hash_convert_tuple(indexRel, values, isnull, values2, isnull2);
I think here, you need to handle the case where heap has a NULL value
as the hash index doesn't contain NULL values, otherwise, the code in
below function can return true which is not right.
4.
+bool
+hashrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
{
..
+ att = indexRel->rd_att->attrs[i - 1];
+ if (!datumIsEqual(values2[i - 1], indxvalue, att->attbyval,
+ att->attlen))
+ {
+ equal = false;
+ break;
+ }
..
}
Hash values are always uint32 and attlen can be different for
different datatypes, so I think above doesn't seem to be the right way
to do the comparison.
5.
@@ -274,6 +301,8 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
OffsetNumber offnum;
ItemPointer current;
bool res;
+ IndexTuple itup;
+
/* Hash
indexes are always lossy since we store only the hash code */
scan->xs_recheck = true;
@@ -316,8
+345,6 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
offnum <=
maxoffnum;
offnum = OffsetNumberNext(offnum))
{
-IndexTuple itup;
-
Why above change?
6.
+ *stats = index_bulk_delete(&ivinfo, *stats,
+lazy_indexvac_phase1, (void *) vacrelstats);
+ ereport(elevel,
+(errmsg("scanned index \"%s\" to remove %d row version, found "
+"%0.f warm pointers, %0.f clear pointers, removed "
+"%0.f warm pointers, removed %0.f clear pointers",
+RelationGetRelationName(indrel),
+ vacrelstats->num_dead_tuples,
+ (*stats)->num_warm_pointers,
+(*stats)->num_clear_pointers,
+(*stats)->warm_pointers_removed,
+ (*stats)->clear_pointers_removed)));
+
+ (*stats)->num_warm_pointers = 0;
+ (*stats)->num_clear_pointers = 0;
+ (*stats)->warm_pointers_removed = 0;
+ (*stats)->clear_pointers_removed = 0;
+ (*stats)->pointers_cleared = 0;
+
+ *stats =index_bulk_delete(&ivinfo, *stats,
+ lazy_indexvac_phase2, (void *)vacrelstats);
To convert WARM chains, we need to do two index passes for all the
indexes. I think it can substantially increase the random I/O. I
think this can help us in doing more WARM updates, but I don't see how
the downside of that (increased random I/O) will be acceptable for all
kind of cases.
+exists. Since index vacuum may visit these pointers in any order, we will need
+another index pass to remove dead index pointers. So in the first index pass we
+check which WARM candidates have 2 index pointers. In the second pass, we
+remove the dead pointer and clear the INDEX_WARM_POINTER flag if that's the
+surviving index pointer.
I think there is some mismatch between README and code. In README, it
is mentioned that dead pointers will be removed in the second phase,
but I think the first phase code lazy_indexvac_phase1() will also
allow to delete the dead pointers (it can return IBDCR_DELETE which
will allow index am to remove dead items.). Am I missing something
here?
7.
+ * For CLEAR chains, we just kill the WARM pointer, if it exist,s and keep
+ * the CLEAR pointer.
typo (exist,s)
8.
+/*
+ * lazy_indexvac_phase2() -- run first pass of index vacuum
Shouldn't this be -- run the second pass
9.
- indexInfo); /* index AM may need this */
+indexInfo, /* index AM may need this */
+(modified_attrs != NULL)); /* type of uniqueness check to do */
comment for the last parameter seems to be wrong.
10.
+follow the update chain everytime to the end to see check if this is a WARM
+chain.
"see check" - seems one of those words is sufficient to explain the meaning.
11.
+chain. This simplifies the design and addresses certain issues around
+duplicate scans.
"duplicate scans" - shouldn't be duplicate key scans.
12.
+index on the table, irrespective of whether the key pertaining to the
+index changed or not.
typo.
/index changed/index is changed
13.
+For example, if we have a table with two columns and two indexes on each
+of the column. When a tuple is first inserted the table, we have exactly
typo.
/inserted the table/inserted in the table
14.
+ lp [1] [2]
+ [1111, aaaa]->[111, bbbb]
Here, after the update, the first column should be 1111.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 11:23 Mithun Cy <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Mithun Cy @ 2017-03-22 11:23 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 3:44 PM, Pavan Deolasee
<[email protected]> wrote:
>
> This looks quite weird to me. Obviously these numbers are completely
> non-comparable. Even the time for VACUUM FULL goes up with every run.
>
> May be we can blame it on AWS instance completely, but the pattern in your
> tests looks very similar where the number slowly and steadily keeps going
> up. If you do complete retest but run v18/v19 first and then run master, may
> be we'll see a complete opposite picture?
>
For those tests I have done tests in the order --- <Master, patch18,
patch18, Master> both the time numbers were same. One different thing
I did was I was deleting the data directory between tests and creating
the database from scratch. Unfortunately the machine I tested this is
not available. I will test same with v19 once I get the machine and
report you back.
--
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 12:30 Greg Stark <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Greg Stark @ 2017-03-22 12:30 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Pavan Deolasee <[email protected]>; Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On 21 March 2017 at 20:04, Bruce Momjian <[email protected]> wrote:
> Yes, but once it is written it will take years before those bits can be
> used on most installations.
Well the problem isn't most installations. On most installations it
should be pretty straightforward to check the oldest database xid and
compare that to when the database was migrated to post-9.0. (Actually
there may be some additional code to write but it's just ensuring that
the bits are actually cleared and not just ignored but even so
databases do generally need to be vacuumed more often than on the
order of years though.)
The problem is that somebody tomorrow could upgrade an 8.4 database to
10.0. In general it seems even versions we don't support get extra
support for migrating away from. I assume it's better to help support
upgrading than to continue to have users using unsupported versions...
And even if you're not concerned about 8.4 someone could still upgrade
9.4 for years to come.
It probably does make sense pick a version, say, 10.0, and have it go
out of its way to ensure it cleans up the MOVED_IN/MOVED_OFF so that
we can be sure that any database was pg_upgraded from 10.0+ doesn't
have any left. Then at least we'll know when the bits are available
again.
--
greg
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 14:20 Mithun Cy <[email protected]>
parent: Pavan Deolasee <[email protected]>
1 sibling, 0 replies; 104+ messages in thread
From: Mithun Cy @ 2017-03-22 14:20 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 8:43 AM, Pavan Deolasee
<[email protected]> wrote:
> Sorry, I did not mean to suggest that you set it up wrongly, I was just
> trying to point out that the test case itself may not be very practical.
That is cool np!, I was just trying to explain why those tests were
made if others wondered about it.
--
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 18:49 Pavan Deolasee <[email protected]>
parent: Mithun Cy <[email protected]>
0 siblings, 3 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-22 18:49 UTC (permalink / raw)
To: Mithun Cy <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy <[email protected]>
wrote:
> On Wed, Mar 22, 2017 at 3:44 PM, Pavan Deolasee
> <[email protected]> wrote:
> >
> > This looks quite weird to me. Obviously these numbers are completely
> > non-comparable. Even the time for VACUUM FULL goes up with every run.
> >
> > May be we can blame it on AWS instance completely, but the pattern in
> your
> > tests looks very similar where the number slowly and steadily keeps going
> > up. If you do complete retest but run v18/v19 first and then run master,
> may
> > be we'll see a complete opposite picture?
> >
>
> For those tests I have done tests in the order --- <Master, patch18,
> patch18, Master> both the time numbers were same.
Hmm, interesting.
> One different thing
> I did was I was deleting the data directory between tests and creating
> the database from scratch. Unfortunately the machine I tested this is
> not available. I will test same with v19 once I get the machine and
> report you back.
Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB
RAM, attached SSD) and results are shown below. But I think it is important
to get independent validation from your side too, just to ensure I am not
making any mistake in measurement. I've attached naively put together
scripts which I used to run the benchmark. If you find them useful, please
adjust the paths and run on your machine.
I reverted back to UNLOGGED table because with WAL the results looked very
weird (as posted earlier) even when I was taking a CHECKPOINT before each
set and had set max_wal_size and checkpoint_timeout high enough to avoid
any checkpoint during the run. Anyways, that's a matter of separate
investigation and not related to this patch.
I did two kinds of tests.
a) update last column of the index
b) update second column of the index
v19 does considerably better than even master for the last column update
case and pretty much inline for the second column update test. The reason
is very clear because v19 determines early in the cycle that the buffer is
already full and there is very little chance of doing a HOT update on the
page. In that case, it does not check any columns for modification. The
master on the other hand will scan through all 9 columns (for last column
update case) and incur the same kind of overhead of doing wasteful work.
The first/second/fourth column shows response time in ms and third and
fifth column shows percentage difference over master. (I hope the table
looks fine, trying some text-table generator tool :-). Apologies if it
looks messed up)
+-------------------------------------------------------+
| Second column update |
+-------------------------------------------------------+
| Master | v18 | v19 |
+-----------+---------------------+---------------------+
| 96657.681 | 108122.868 | 11.86% | 96873.642 | 0.22% |
+-----------+------------+--------+------------+--------+
| 98546.35 | 110021.27 | 11.64% | 97569.187 | -0.99% |
+-----------+------------+--------+------------+--------+
| 99297.231 | 110550.054 | 11.33% | 100241.261 | 0.95% |
+-----------+------------+--------+------------+--------+
| 97196.556 | 110235.808 | 13.42% | 97216.207 | 0.02% |
+-----------+------------+--------+------------+--------+
| 99072.432 | 110477.327 | 11.51% | 97950.687 | -1.13% |
+-----------+------------+--------+------------+--------+
| 96730.056 | 109217.939 | 12.91% | 96929.617 | 0.21% |
+-----------+------------+--------+------------+--------+
+-------------------------------------------------------+
| Last column update |
+-------------------------------------------------------+
| Master | v18 | v19 |
+------------+--------------------+---------------------+
| 112545.537 | 116563.608 | 3.57% | 103067.276 | -8.42% |
+------------+------------+-------+------------+--------+
| 110169.224 | 115753.991 | 5.07% | 104411.44 | -5.23% |
+------------+------------+-------+------------+--------+
| 112280.874 | 116437.11 | 3.70% | 104868.98 | -6.60% |
+------------+------------+-------+------------+--------+
| 113171.556 | 116813.262 | 3.22% | 103907.012 | -8.19% |
+------------+------------+-------+------------+--------+
| 110721.881 | 117442.709 | 6.07% | 104124.131 | -5.96% |
+------------+------------+-------+------------+--------+
| 112138.601 | 115834.549 | 3.30% | 104858.624 | -6.49% |
+------------+------------+-------+------------+--------+
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gzip] interest_attrs_tests.tar.gz (7.8K, ../../CABOikdNDDO_jH3XCTy5zKfK6z=YisbqDFQc0Oz6GXKkdCdyyuA@mail.gmail.com/3-interest_attrs_tests.tar.gz)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-22 19:17 Mithun Cy <[email protected]>
parent: Pavan Deolasee <[email protected]>
2 siblings, 0 replies; 104+ messages in thread
From: Mithun Cy @ 2017-03-22 19:17 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee
<[email protected]> wrote:
>
>
> On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy <[email protected]>
> wrote:
> Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB
> RAM, attached SSD) and results are shown below. But I think it is important
> to get independent validation from your side too, just to ensure I am not
> making any mistake in measurement. I've attached naively put together
> scripts which I used to run the benchmark. If you find them useful, please
> adjust the paths and run on your machine.
Looking at your postgresql.conf JFYI, I have synchronous_commit = off
but same is on with your run (for logged tables) and rest remains
same. Once I get the machine probably next morning, I will run same
tests on v19.
--
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 04:58 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Amit Kapila @ 2017-03-23 04:58 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila <[email protected]> wrote:
> On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee
> <[email protected]> wrote:
>>
>>>
>>
>> Please find attached rebased patches.
>>
>
> Few comments on 0005_warm_updates_v18.patch:
>
Few more comments on 0005_warm_updates_v18.patch:
1.
@@ -234,6 +241,25 @@ index_beginscan(Relation heapRelation,
scan->heapRelation = heapRelation;
scan->xs_snapshot = snapshot;
+ /*
+ * If the index supports recheck,
make sure that index tuple is saved
+ * during index scans. Also build and cache IndexInfo which is used by
+ * amrecheck routine.
+ *
+ * XXX Ideally, we should look at
all indexes on the table and check if
+ * WARM is at all supported on the base table. If WARM is not supported
+ * then we don't need to do any recheck.
RelationGetIndexAttrBitmap() does
+ * do that and sets rd_supportswarm after looking at all indexes. But we
+ * don't know if the function was called earlier in the
session when we're
+ * here. We can't call it now because there exists a risk of causing
+ * deadlock.
+ */
+ if (indexRelation->rd_amroutine->amrecheck)
+ {
+scan->xs_want_itup = true;
+ scan->indexInfo = BuildIndexInfo(indexRelation);
+ }
+
Don't we need to do this rechecking during parallel scans? Also what
about bitmap heap scans?
2.
+++ b/src/backend/access/nbtree/nbtinsert.c
-
typedef struct
Above change is not require.
3.
+_bt_clear_items(Page page, OffsetNumber *clearitemnos, uint16 nclearitems)
+void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems)
Both the above functions look exactly same, isn't it better to have a
single function like page_clear_items? If you want separation for
different index types, then we can have one common function which can
be called from different index types.
4.
- if (callback(htup, callback_state))
+ flags = ItemPointerGetFlags(&itup->t_tid);
+ is_warm = ((flags & BTREE_INDEX_WARM_POINTER) != 0);
+
+ if (is_warm)
+ stats->num_warm_pointers++;
+ else
+ stats->num_clear_pointers++;
+
+ result = callback(htup, is_warm, callback_state);
+ if (result == IBDCR_DELETE)
+ {
+ if (is_warm)
+ stats->warm_pointers_removed++;
+ else
+ stats->clear_pointers_removed++;
The patch looks to be inconsistent in collecting stats for btree and
hash. I don't see above stats are getting updated in hash index code.
5.
+btrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
{
..
+ if (!datumIsEqual(values[i - 1], indxvalue, att->attbyval,
+ att->attlen))
..
}
Will this work if the index is using non-default collation?
6.
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -390,83 +390,9 @@ btree_xlog_vacuum(XLogReaderState *record)
-#ifdef UNUSED
xl_btree_vacuum *xlrec = (xl_btree_vacuum *) XLogRecGetData(record);
/*
- * This section of code is thought to be no longer needed, after analysis
- * of the calling paths. It is retained to allow the code to be reinstated
- * if a flaw is revealed in that thinking.
- *
..
Why does this patch need to remove the above code under #ifdef UNUSED
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 09:32 Amit Kapila <[email protected]>
parent: Pavan Deolasee <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Amit Kapila @ 2017-03-23 09:32 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Mithun Cy <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee
<[email protected]> wrote:
>
> Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB
> RAM, attached SSD) and results are shown below. But I think it is important
> to get independent validation from your side too, just to ensure I am not
> making any mistake in measurement. I've attached naively put together
> scripts which I used to run the benchmark. If you find them useful, please
> adjust the paths and run on your machine.
>
> I reverted back to UNLOGGED table because with WAL the results looked very
> weird (as posted earlier) even when I was taking a CHECKPOINT before each
> set and had set max_wal_size and checkpoint_timeout high enough to avoid any
> checkpoint during the run. Anyways, that's a matter of separate
> investigation and not related to this patch.
>
> I did two kinds of tests.
> a) update last column of the index
> b) update second column of the index
>
> v19 does considerably better than even master for the last column update
> case and pretty much inline for the second column update test. The reason is
> very clear because v19 determines early in the cycle that the buffer is
> already full and there is very little chance of doing a HOT update on the
> page. In that case, it does not check any columns for modification.
>
That sounds like you are dodging the actual problem. I mean you can
put that same PageIsFull() check in master code as well and then you
will most probably again see the same regression. Also, I think if we
test at fillfactor 80 or 75 (which is not unrealistic considering an
update-intensive workload), then we might again see regression.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 10:14 Pavan Deolasee <[email protected]>
parent: Amit Kapila <[email protected]>
1 sibling, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 10:14 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila <[email protected]>
wrote:
> On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee
> <[email protected]> wrote:
> >
> >>
> >
> > Please find attached rebased patches.
> >
>
> Few comments on 0005_warm_updates_v18.patch:
>
>
Thanks a lot Amit for review comments.
> 1.
> @@ -806,20 +835,35 @@ hashbucketcleanup(Relation rel, Bucket
> cur_bucket, Buffer bucket_buf,
> {
> ..
> - if (callback && callback(htup, callback_state))
> + if(callback)
> {
> - kill_tuple = true;
> -
> if (tuples_removed)
> - *tuples_removed += 1;
> +result = callback(htup, is_warm, callback_state);
> + if (result== IBDCR_DELETE)
> + {
> + kill_tuple = true;
> + if (tuples_removed)
> +*tuples_removed += 1;
> + }
> + else if (result ==IBDCR_CLEAR_WARM)
> + {
> + clear_tuple= true;
> + }
> }
> else if
> (split_cleanup)
> ..
> }
>
> I think this will break the existing mechanism of split cleanup. We
> need to check for split cleanup if the tuple is tuple is not deletable
> by the callback. This is not merely an optimization but a must
> condition because we will clear the split cleanup flag after this
> bucket is scanned completely.
>
>
Ok, I see. Fixed, but please check if this looks good.
> 2.
> - PageIndexMultiDelete(page, deletable, ndeletable);
> + /*
> +
> * Clear the WARM pointers.
> + *
> + * We mustdo this before dealing with the dead items because
> + * PageIndexMultiDelete may move items around to compactify the
> + * array and hence offnums recorded earlierwon't make any sense
> + * after PageIndexMultiDelete is called.
> +
> */
> + if (nclearwarm > 0)
> + _hash_clear_items(page,clearwarm, nclearwarm);
> +
> + /*
> + * And delete the deletableitems
> + */
> + if (ndeletable > 0)
> +
> PageIndexMultiDelete(page, deletable, ndeletable);
>
> I think this assumes that the items where we need to clear warm flag
> are not deletable, otherwise what is the need to clear the flag if we
> are going to delete the tuple. The deletable tuple can have a warm
> flag if it is deletable due to split cleanup.
>
>
Yes. Since the callback will either say IBDCR_DELETE or IBDCR_CLEAR_WARM, I
don't think we will ever has a situation where a tuple is deleted as well
as cleared. I also checked that the bucket split code should carry the WARM
flag correctly to the new bucket.
Based on your first comment, I believe the rearranged code with take care
of deleting a tuple even if WARM flag is set, if the deletion is because of
bucket split.
3.
> + /*
> + * HASH indexes compute a hash value of the key and store that in the
> + * index. So
> we must first obtain the hash of the value obtained from the
> + * heap and then do a comparison
> +
> */
> + _hash_convert_tuple(indexRel, values, isnull, values2, isnull2);
>
> I think here, you need to handle the case where heap has a NULL value
> as the hash index doesn't contain NULL values, otherwise, the code in
> below function can return true which is not right.
>
>
I think we can simply conclude hashrecheck has failed the equality if the
heap has NULL value because such a tuple should not have been reached via
hash index unless a non-NULL hash key was later updated to a NULL key,
right?
> 4.
> +bool
> +hashrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple
> indexTuple,
> + Relation heapRel, HeapTuple heapTuple)
> {
> ..
> + att = indexRel->rd_att->attrs[i - 1];
> + if (!datumIsEqual(values2[i - 1], indxvalue, att->attbyval,
> + att->attlen))
> + {
> + equal = false;
> + break;
> + }
> ..
> }
>
> Hash values are always uint32 and attlen can be different for
> different datatypes, so I think above doesn't seem to be the right way
> to do the comparison.
>
>
Since we're referring to the attr from the index relation, wouldn't it tell
us the attribute specs of what gets stored in the index and not what's
there in the heap? I could be wrong but some quick tests show me that
pg_attribute->attlen for the index relation always returns 4 irrespective
of the underlying data type in heap.
> 5.
> @@ -274,6 +301,8 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
> OffsetNumber offnum;
>
> ItemPointer current;
> bool res;
> + IndexTuple itup;
> +
>
> /* Hash
> indexes are always lossy since we store only the hash code */
> scan->xs_recheck = true;
> @@ -316,8
> +345,6 @@ hashgettuple(IndexScanDesc scan, ScanDirection dir)
> offnum <=
> maxoffnum;
> offnum = OffsetNumberNext(offnum))
> {
> -IndexTuple itup;
> -
>
> Why above change?
>
>
Seems spurious. Fixed.
>
> 6.
> + *stats = index_bulk_delete(&ivinfo, *stats,
> +lazy_indexvac_phase1, (void *) vacrelstats);
> + ereport(elevel,
> +(errmsg("scanned index \"%s\" to remove %d row version, found "
> +"%0.f warm pointers, %0.f clear pointers, removed "
> +"%0.f warm pointers, removed %0.f clear pointers",
> +RelationGetRelationName(indrel),
> + vacrelstats->num_dead_tuples,
> + (*stats)->num_warm_pointers,
> +(*stats)->num_clear_pointers,
> +(*stats)->warm_pointers_removed,
> + (*stats)->clear_pointers_removed)));
> +
> + (*stats)->num_warm_pointers = 0;
> + (*stats)->num_clear_pointers = 0;
> + (*stats)->warm_pointers_removed = 0;
> + (*stats)->clear_pointers_removed = 0;
> + (*stats)->pointers_cleared = 0;
> +
> + *stats =index_bulk_delete(&ivinfo, *stats,
> + lazy_indexvac_phase2, (void *)vacrelstats);
>
> To convert WARM chains, we need to do two index passes for all the
> indexes. I think it can substantially increase the random I/O. I
> think this can help us in doing more WARM updates, but I don't see how
> the downside of that (increased random I/O) will be acceptable for all
> kind of cases.
>
>
Yes, this is a very fair point. The way I proposed to address this upthread
is by introducing a set of threshold/scale GUCs specific to WARM. So users
can control when to invoke WARM cleanup. Only if the WARM cleanup is
required, we do 2 index scans. Otherwise vacuum will work the way it works
today without any additional overhead.
We already have some intelligence to skip the second index scan if we did
not find any WARM candidate chains during the first heap scan. This should
take care of majority of the users who never update their indexed columns.
For others, we need either a knob or some built-in way to deduce whether to
do WARM cleanup or not.
Does that seem worthwhile?
>
> +exists. Since index vacuum may visit these pointers in any order, we will
> need
> +another index pass to remove dead index pointers. So in the first index
> pass we
> +check which WARM candidates have 2 index pointers. In the second pass, we
> +remove the dead pointer and clear the INDEX_WARM_POINTER flag if that's
> the
> +surviving index pointer.
>
> I think there is some mismatch between README and code. In README, it
> is mentioned that dead pointers will be removed in the second phase,
> but I think the first phase code lazy_indexvac_phase1() will also
> allow to delete the dead pointers (it can return IBDCR_DELETE which
> will allow index am to remove dead items.). Am I missing something
> here?
>
>
Hmm.. fixed the README. Clearly we do allow removal of dead pointers which
are known to be certainly dead in the first index pass itself. Some other
pointers can be removed during the second scan once we know about the
existence or non existence of WARM index pointers.
>
> 7.
> + * For CLEAR chains, we just kill the WARM pointer, if it exist,s and keep
> + * the CLEAR pointer.
>
> typo (exist,s)
>
>
Fixed.
> 8.
> +/*
> + * lazy_indexvac_phase2() -- run first pass of index vacuum
>
> Shouldn't this be -- run the second pass
>
>
Yes, fixed.
> 9.
> - indexInfo); /* index AM may need this */
> +indexInfo, /* index AM may need this */
> +(modified_attrs != NULL)); /* type of uniqueness check to do */
>
> comment for the last parameter seems to be wrong.
>
>
Fixed.
> 10.
> +follow the update chain everytime to the end to see check if this is a
> WARM
> +chain.
>
> "see check" - seems one of those words is sufficient to explain the
> meaning.
>
>
Fixed.
> 11.
> +chain. This simplifies the design and addresses certain issues around
> +duplicate scans.
>
> "duplicate scans" - shouldn't be duplicate key scans.
>
>
Ok, seems better. Fixed.
> 12.
> +index on the table, irrespective of whether the key pertaining to the
> +index changed or not.
>
> typo.
> /index changed/index is changed
>
>
Fixed.
> 13.
> +For example, if we have a table with two columns and two indexes on each
> +of the column. When a tuple is first inserted the table, we have exactly
>
> typo.
> /inserted the table/inserted in the table
>
>
Fixed.
> 14.
> + lp [1] [2]
> + [1111, aaaa]->[111, bbbb]
>
> Here, after the update, the first column should be 1111.
>
>
Fixed.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0005_warm_updates_v19.patch (216.5K, ../../CABOikdPEhYdsDqa07QR+x-TAeBsaNdSUNoosSzo8xTQ79W-UUw@mail.gmail.com/3-0005_warm_updates_v19.patch)
download | inline diff:
diff --git b/contrib/bloom/blutils.c a/contrib/bloom/blutils.c
index f2eda67..b356e2b 100644
--- b/contrib/bloom/blutils.c
+++ a/contrib/bloom/blutils.c
@@ -142,6 +142,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git b/contrib/bloom/blvacuum.c a/contrib/bloom/blvacuum.c
index 04abd0f..ff50361 100644
--- b/contrib/bloom/blvacuum.c
+++ a/contrib/bloom/blvacuum.c
@@ -88,7 +88,7 @@ blbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
while (itup < itupEnd)
{
/* Do we have to delete this tuple? */
- if (callback(&itup->heapPtr, callback_state))
+ if (callback(&itup->heapPtr, false, callback_state) == IBDCR_DELETE)
{
/* Yes; adjust count of tuples that will be left on page */
BloomPageGetOpaque(page)->maxoff--;
diff --git b/src/backend/access/brin/brin.c a/src/backend/access/brin/brin.c
index b22563b..b4a1465 100644
--- b/src/backend/access/brin/brin.c
+++ a/src/backend/access/brin/brin.c
@@ -116,6 +116,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git b/src/backend/access/gin/ginvacuum.c a/src/backend/access/gin/ginvacuum.c
index c9ccfee..8ed71c5 100644
--- b/src/backend/access/gin/ginvacuum.c
+++ a/src/backend/access/gin/ginvacuum.c
@@ -56,7 +56,8 @@ ginVacuumItemPointers(GinVacuumState *gvs, ItemPointerData *items,
*/
for (i = 0; i < nitem; i++)
{
- if (gvs->callback(items + i, gvs->callback_state))
+ if (gvs->callback(items + i, false, gvs->callback_state) ==
+ IBDCR_DELETE)
{
gvs->result->tuples_removed += 1;
if (!tmpitems)
diff --git b/src/backend/access/gist/gist.c a/src/backend/access/gist/gist.c
index 6593771..843389b 100644
--- b/src/backend/access/gist/gist.c
+++ a/src/backend/access/gist/gist.c
@@ -94,6 +94,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git b/src/backend/access/gist/gistvacuum.c a/src/backend/access/gist/gistvacuum.c
index 77d9d12..0955db6 100644
--- b/src/backend/access/gist/gistvacuum.c
+++ a/src/backend/access/gist/gistvacuum.c
@@ -202,7 +202,8 @@ gistbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
iid = PageGetItemId(page, i);
idxtuple = (IndexTuple) PageGetItem(page, iid);
- if (callback(&(idxtuple->t_tid), callback_state))
+ if (callback(&(idxtuple->t_tid), false, callback_state) ==
+ IBDCR_DELETE)
todelete[ntodelete++] = i;
else
stats->num_index_tuples += 1;
diff --git b/src/backend/access/hash/hash.c a/src/backend/access/hash/hash.c
index cfcec34..a930a1c 100644
--- b/src/backend/access/hash/hash.c
+++ a/src/backend/access/hash/hash.c
@@ -75,6 +75,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambuild = hashbuild;
amroutine->ambuildempty = hashbuildempty;
amroutine->aminsert = hashinsert;
+ amroutine->amwarminsert = hashwarminsert;
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
@@ -92,6 +93,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = hashrecheck;
PG_RETURN_POINTER(amroutine);
}
@@ -233,11 +235,11 @@ hashbuildCallback(Relation index,
* Hash on the heap tuple's key, form an index tuple with hash code.
* Find the appropriate location for the new tuple, and put it there.
*/
-bool
-hashinsert(Relation rel, Datum *values, bool *isnull,
+static bool
+hashinsert_internal(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo, bool warm_update)
{
Datum index_values[1];
bool index_isnull[1];
@@ -253,6 +255,11 @@ hashinsert(Relation rel, Datum *values, bool *isnull,
itup = index_form_tuple(RelationGetDescr(rel), index_values, index_isnull);
itup->t_tid = *ht_ctid;
+ if (warm_update)
+ ItemPointerSetFlags(&itup->t_tid, HASH_INDEX_WARM_POINTER);
+ else
+ ItemPointerClearFlags(&itup->t_tid);
+
_hash_doinsert(rel, itup, heapRel);
pfree(itup);
@@ -260,6 +267,26 @@ hashinsert(Relation rel, Datum *values, bool *isnull,
return false;
}
+bool
+hashinsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return hashinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, false);
+}
+
+bool
+hashwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return hashinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, true);
+
+}
/*
* hashgettuple() -- Get the next tuple in the scan.
@@ -536,6 +563,9 @@ hashbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
Relation rel = info->index;
double tuples_removed;
double num_index_tuples;
+ double warm_pointers_removed;
+ double clear_pointers_removed;
+ double pointers_cleared;
double orig_ntuples;
Bucket orig_maxbucket;
Bucket cur_maxbucket;
@@ -544,7 +574,8 @@ hashbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
HashMetaPage metap;
HashMetaPage cachedmetap;
- tuples_removed = 0;
+ tuples_removed = warm_pointers_removed = clear_pointers_removed = 0;
+ pointers_cleared = 0;
num_index_tuples = 0;
/*
@@ -627,7 +658,9 @@ loop_top:
cachedmetap->hashm_maxbucket,
cachedmetap->hashm_highmask,
cachedmetap->hashm_lowmask, &tuples_removed,
- &num_index_tuples, split_cleanup,
+ &num_index_tuples, &warm_pointers_removed,
+ &clear_pointers_removed, &pointers_cleared,
+ split_cleanup,
callback, callback_state);
_hash_dropbuf(rel, bucket_buf);
@@ -709,6 +742,8 @@ loop_top:
stats->estimated_count = false;
stats->num_index_tuples = num_index_tuples;
stats->tuples_removed += tuples_removed;
+ stats->warm_pointers_removed += warm_pointers_removed;
+ stats->clear_pointers_removed += clear_pointers_removed;
/* hashvacuumcleanup will fill in num_pages */
return stats;
@@ -764,6 +799,9 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
BlockNumber bucket_blkno, BufferAccessStrategy bstrategy,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
+ double *warm_pointers_removed,
+ double *clear_pointers_removed,
+ double *pointers_cleared,
bool split_cleanup,
IndexBulkDeleteCallback callback, void *callback_state)
{
@@ -789,6 +827,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
Page page;
OffsetNumber deletable[MaxOffsetNumber];
int ndeletable = 0;
+ OffsetNumber clearwarm[MaxOffsetNumber];
+ int nclearwarm = 0;
bool retain_pin = false;
vacuum_delay_point();
@@ -806,22 +846,44 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
IndexTuple itup;
Bucket bucket;
bool kill_tuple = false;
+ bool clear_tuple = false;
+ int flags;
+ bool is_warm;
+ IndexBulkDeleteCallbackResult result;
itup = (IndexTuple) PageGetItem(page,
PageGetItemId(page, offno));
htup = &(itup->t_tid);
+ flags = ItemPointerGetFlags(&itup->t_tid);
+ is_warm = ((flags & HASH_INDEX_WARM_POINTER) != 0);
+
/*
* To remove the dead tuples, we strictly want to rely on results
* of callback function. refer btvacuumpage for detailed reason.
*/
- if (callback && callback(htup, callback_state))
+ if (callback)
{
- kill_tuple = true;
- if (tuples_removed)
- *tuples_removed += 1;
+ result = callback(htup, is_warm, callback_state);
+ if (result == IBDCR_DELETE)
+ {
+ kill_tuple = true;
+ if (tuples_removed)
+ *tuples_removed += 1;
+ if (is_warm && warm_pointers_removed)
+ *warm_pointers_removed += 1;
+ else if (!is_warm && clear_pointers_removed)
+ *clear_pointers_removed += 1;
+ }
+ else if (result == IBDCR_CLEAR_WARM)
+ {
+ clear_tuple = true;
+ if (pointers_cleared)
+ *pointers_cleared += 1;
+ }
}
- else if (split_cleanup)
+
+ if (!kill_tuple && split_cleanup)
{
/* delete the tuples that are moved by split. */
bucket = _hash_hashkey2bucket(_hash_get_indextuple_hashkey(itup),
@@ -849,6 +911,9 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
}
else
{
+ if (clear_tuple)
+ /* clear the WARM pointer */
+ clearwarm[nclearwarm++] = offno;
/* we're keeping it, so count it */
if (num_index_tuples)
*num_index_tuples += 1;
@@ -866,12 +931,27 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
/*
* Apply deletions, advance to next page and write page if needed.
*/
- if (ndeletable > 0)
+ if (ndeletable > 0 || nclearwarm > 0)
{
/* No ereport(ERROR) until changes are logged */
START_CRIT_SECTION();
- PageIndexMultiDelete(page, deletable, ndeletable);
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the
+ * array and hence offnums recorded earlier won't make any sense
+ * after PageIndexMultiDelete is called.
+ */
+ if (nclearwarm > 0)
+ _hash_clear_items(page, clearwarm, nclearwarm);
+
+ /*
+ * And delete the deletable items
+ */
+ if (ndeletable > 0)
+ PageIndexMultiDelete(page, deletable, ndeletable);
bucket_dirty = true;
/*
@@ -892,6 +972,7 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
XLogRecPtr recptr;
xlrec.is_primary_bucket_page = (buf == bucket_buf) ? true : false;
+ xlrec.nclearitems = nclearwarm;
XLogBeginInsert();
XLogRegisterData((char *) &xlrec, SizeOfHashDelete);
@@ -904,6 +985,8 @@ hashbucketcleanup(Relation rel, Bucket cur_bucket, Buffer bucket_buf,
XLogRegisterBuffer(0, bucket_buf, REGBUF_STANDARD | REGBUF_NO_IMAGE);
XLogRegisterBuffer(1, buf, REGBUF_STANDARD);
+ XLogRegisterBufData(1, (char *) clearwarm,
+ nclearwarm * sizeof(OffsetNumber));
XLogRegisterBufData(1, (char *) deletable,
ndeletable * sizeof(OffsetNumber));
diff --git b/src/backend/access/hash/hash_xlog.c a/src/backend/access/hash/hash_xlog.c
index 8647e8c..fe89ee1 100644
--- b/src/backend/access/hash/hash_xlog.c
+++ a/src/backend/access/hash/hash_xlog.c
@@ -840,6 +840,7 @@ hash_xlog_delete(XLogReaderState *record)
/* replay the record for deleting entries in bucket page */
if (action == BLK_NEEDS_REDO)
{
+ uint16 nclearwarm = xldata->nclearitems;
char *ptr;
Size len;
@@ -849,12 +850,17 @@ hash_xlog_delete(XLogReaderState *record)
if (len > 0)
{
+ OffsetNumber *clearwarm;
OffsetNumber *unused;
OffsetNumber *unend;
- unused = (OffsetNumber *) ptr;
+ clearwarm = (OffsetNumber *) ptr;
+ unused = clearwarm + nclearwarm;
unend = (OffsetNumber *) ((char *) ptr + len);
+ if (nclearwarm)
+ _hash_clear_items(page, clearwarm, nclearwarm);
+
if ((unend - unused) > 0)
PageIndexMultiDelete(page, unused, unend - unused);
}
diff --git b/src/backend/access/hash/hashpage.c a/src/backend/access/hash/hashpage.c
index 622cc4b..c349b28 100644
--- b/src/backend/access/hash/hashpage.c
+++ a/src/backend/access/hash/hashpage.c
@@ -754,8 +754,8 @@ restart_expand:
LockBuffer(metabuf, BUFFER_LOCK_UNLOCK);
hashbucketcleanup(rel, old_bucket, buf_oblkno, start_oblkno, NULL,
- maxbucket, highmask, lowmask, NULL, NULL, true,
- NULL, NULL);
+ maxbucket, highmask, lowmask, NULL, NULL, NULL,
+ NULL, NULL, true, NULL, NULL);
_hash_dropbuf(rel, buf_oblkno);
@@ -1576,3 +1576,13 @@ _hash_getbucketbuf_from_hashkey(Relation rel, uint32 hashkey, int access,
return buf;
}
+
+/*
+ * Currently just a wrapper arounf PageIndexClearWarmTuples, but in theory each
+ * index may have it's own way to handle WARM tuples.
+ */
+void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems)
+{
+ PageIndexClearWarmTuples(page, clearitemnos, nclearitems);
+}
diff --git b/src/backend/access/hash/hashsearch.c a/src/backend/access/hash/hashsearch.c
index 2d92049..330ccc5 100644
--- b/src/backend/access/hash/hashsearch.c
+++ a/src/backend/access/hash/hashsearch.c
@@ -59,6 +59,8 @@ _hash_next(IndexScanDesc scan, ScanDirection dir)
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
so->hashso_heappos = itup->t_tid;
+ if (scan->xs_want_itup)
+ scan->xs_itup = itup;
return true;
}
@@ -367,6 +369,9 @@ _hash_first(IndexScanDesc scan, ScanDirection dir)
itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, offnum));
so->hashso_heappos = itup->t_tid;
+ if (scan->xs_want_itup)
+ scan->xs_itup = itup;
+
return true;
}
diff --git b/src/backend/access/hash/hashutil.c a/src/backend/access/hash/hashutil.c
index 2e99719..caac8a9 100644
--- b/src/backend/access/hash/hashutil.c
+++ a/src/backend/access/hash/hashutil.c
@@ -17,9 +17,11 @@
#include "access/hash.h"
#include "access/reloptions.h"
#include "access/relscan.h"
+#include "catalog/index.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "storage/buf_internals.h"
+#include "utils/datum.h"
#define CALC_NEW_BUCKET(old_bucket, lowmask) \
old_bucket | (lowmask + 1)
@@ -514,3 +516,77 @@ _hash_kill_items(IndexScanDesc scan)
MarkBufferDirtyHint(so->hashso_curbuf, true);
}
}
+
+/*
+ * Recheck if the heap tuple satisfies the key stored in the index tuple
+ */
+bool
+hashrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
+{
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ bool isavail[INDEX_MAX_KEYS];
+ Datum values2[INDEX_MAX_KEYS];
+ bool isnull2[INDEX_MAX_KEYS];
+ int i;
+ bool equal;
+ int natts = indexRel->rd_rel->relnatts;
+ Form_pg_attribute att;
+
+ FormIndexPlainDatum(indexInfo, heapRel, heapTuple, values, isnull, isavail);
+
+ /*
+ * HASH indexes compute a hash value of the key and store that in the
+ * index. So we must first obtain the hash of the value obtained from the
+ * heap and then do a comparison.
+ *
+ * If the user-supplied value is NULL then we can just assume that the
+ * recheck has failed because the hash index must not have inserted any
+ * entry for a NULL-value to start with. So if heap is presenting a NULL
+ * value now, that pretty much means that the heap contains an updated
+ * value not satisfying the hash index key.
+ */
+ if (!_hash_convert_tuple(indexRel, values, isnull, values2, isnull2))
+ return false;
+
+ equal = true;
+ for (i = 1; i <= natts; i++)
+ {
+ Datum indxvalue;
+ bool indxisnull;
+
+ if (!isavail[i - 1])
+ continue;
+
+ indxvalue = index_getattr(indexTuple, i, indexRel->rd_att, &indxisnull);
+
+ /*
+ * If both are NULL then they are equal
+ */
+ if (isnull2[i - 1] && indxisnull)
+ continue;
+
+ /*
+ * If either is NULL then they are not equal
+ */
+ if (isnull2[i - 1] || indxisnull)
+ {
+ equal = false;
+ break;
+ }
+
+ /*
+ * Now do a raw memory comparison
+ */
+ att = indexRel->rd_att->attrs[i - 1];
+ if (!datumIsEqual(values2[i - 1], indxvalue, att->attbyval,
+ att->attlen))
+ {
+ equal = false;
+ break;
+ }
+ }
+
+ return equal;
+}
diff --git b/src/backend/access/heap/README.WARM a/src/backend/access/heap/README.WARM
new file mode 100644
index 0000000..7c93a70
--- /dev/null
+++ a/src/backend/access/heap/README.WARM
@@ -0,0 +1,308 @@
+src/backend/access/heap/README.WARM
+
+Write Amplification Reduction Method (WARM)
+===========================================
+
+The Heap Only Tuple (HOT) feature greatly eliminated redudant index
+entries and allowed re-use of the dead space occupied by previously
+updated or deleted tuples (see src/backend/access/heap/README.HOT)
+
+One of the necessary conditions for satisfying HOT update is that the
+update must not change a column used in any of the indexes on the table.
+The condition is sometimes hard to meet, especially for complex
+workloads with several indexes on large yet frequently updated tables.
+Worse, sometimes only one or two index columns may be updated, but the
+regular non-HOT update will still insert a new index entry in every
+index on the table, irrespective of whether the key pertaining to the
+index is changed or not.
+
+WARM is a technique devised to address these problems.
+
+
+Update Chains With Multiple Index Entries Pointing to the Root
+--------------------------------------------------------------
+
+When a non-HOT update is caused by an index key change, a new index
+entry must be inserted for the changed index. But if the index key
+hasn't changed for other indexes, we don't really need to insert a new
+entry. Even though the existing index entry is pointing to the old
+tuple, the new tuple is reachable via the t_ctid chain. To keep things
+simple, a WARM update requires that the heap block must have enough
+space to store the new version of the tuple. This is same as HOT
+updates.
+
+In WARM, we ensure that every index entry always points to the root of
+the WARM chain. In fact, a WARM chain looks exactly like a HOT chain
+except for the fact that there could be multiple index entries pointing
+to the root of the chain. So when new entry is inserted in an index for
+updated tuple, and if we are doing a WARM update, the new entry is made
+point to the root of the WARM chain.
+
+For example, if we have a table with two columns and two indexes on each of the
+column. When a tuple is first inserted into the table, we have exactly one
+index entry pointing to the tuple from both indexes.
+
+ lp [1]
+ [1111, aaaa]
+
+ Index1's entry (1111) points to 1
+ Index2's entry (aaaa) also points to 1
+
+Now if the tuple's second column is updated and if there is room on the
+page, we perform a WARM update. To do so, Index1 does not get any new
+entry and Index2's new entry will still point to the root tuple of the
+chain.
+
+ lp [1] [2]
+ [1111, aaaa]->[1111, bbbb]
+
+ Index1's entry (1111) points to 1
+ Index2's old entry (aaaa) points to 1
+ Index2's new entry (bbbb) also points to 1
+
+"A update chain which has more than one index entries pointing to its
+root line pointer is called WARM chain and the action that creates a
+WARM chain is called WARM update."
+
+Since all indexes always point to the root of the WARM chain, even when
+there are more than one index entries, WARM chains can be pruned and
+dead tuples can be removed without a need to do corresponding index
+cleanup.
+
+While this solves the problem of pruning dead tuples from a HOT/WARM
+chain, it also opens up a new technical challenge because now we have a
+situation where a heap tuple is reachable from multiple index entries,
+each having a different index key. While MVCC still ensures that only
+valid tuples are returned, a tuple with a wrong index key may be
+returned because of wrong index entries. In the above example, tuple
+[1111, bbbb] is reachable from both keys (aaaa) as well as (bbbb). For
+this reason, tuples returned from a WARM chain must always be rechecked
+for index key-match.
+
+Recheck Index Key Againt Heap Tuple
+-----------------------------------
+
+Since every Index AM has it's own notion of index tuples, each Index AM
+must implement its own method to recheck heap tuples. For example, a
+hash index stores the hash value of the column and hence recheck routine
+for hash AM must first compute the hash value of the heap attribute and
+then compare it against the value stored in the index tuple.
+
+The patch currently implement recheck routines for hash and btree
+indexes. If the table has an index which doesn't support recheck
+routine, WARM updates are disabled on such tables.
+
+Problem With Duplicate (key, ctid) Index Entries
+------------------------------------------------
+
+The index-key recheck logic works as long as there are no duplicate
+index keys, both pointing to the same WARM chain. In that case, the same
+valid tuple will be reachable via multiple index keys, yet satisfying
+the index key checks. In the above example, if the tuple [1111, bbbb] is
+again updated to [1111, aaaa] and if we insert a new index entry (aaaa)
+pointing to the root line pointer, we will end up with the following
+structure:
+
+ lp [1] [2] [3]
+ [1111, aaaa]->[1111, bbbb]->[1111, aaaa]
+
+ Index1's entry (1111) points to 1
+ Index2's oldest entry (aaaa) points to 1
+ Index2's old entry (bbbb) also points to 1
+ Index2's new entry (aaaa) also points to 1
+
+We must solve this problem to ensure that the same tuple is not
+reachable via multiple index pointers. There are couple of ways to
+address this issue:
+
+1. Do not allow WARM update to a tuple from a WARM chain. This
+guarantees that there can never be duplicate index entries to the same
+root line pointer because we must have checked for old and new index
+keys while doing the first WARM update.
+
+2. Do not allow duplicate (key, ctid) index pointers. In the above
+example, since (aaaa, 1) already exists in the index, we must not insert
+a duplicate index entry.
+
+The patch currently implements 1 i.e. do not do WARM updates to a tuple
+from a WARM chain. HOT updates are fine because they do not add a new
+index entry.
+
+Even with the restriction, this is a significant improvement because the
+number of regular UPDATEs are curtailed down to half.
+
+Expression and Partial Indexes
+------------------------------
+
+Expressions may evaluate to the same value even if the underlying column
+values have changed. A simple example is an index on "lower(col)" which
+will return the same value if the new heap value only differs in the
+case sensitivity. So we can not solely rely on the heap column check to
+decide whether or not to insert a new index entry for expression
+indexes. Similarly, for partial indexes, the predicate expression must
+be evaluated to decide whether or not to cause a new index entry when
+columns referred in the predicate expressions change.
+
+(None of these things are currently implemented and we squarely disallow
+WARM update if a column from expression indexes or predicate has
+changed).
+
+
+Efficiently Finding the Root Line Pointer
+-----------------------------------------
+
+During WARM update, we must be able to find the root line pointer of the
+tuple being updated. It must be noted that the t_ctid field in the heap
+tuple header is usually used to find the next tuple in the update chain.
+But the tuple that we are updating, must be the last tuple in the update
+chain. In such cases, the c_tid field usually points the tuple itself.
+So in theory, we could use the t_ctid to store additional information in
+the last tuple of the update chain, if the information about the tuple
+being the last tuple is stored elsewhere.
+
+We now utilize another bit from t_infomask2 to explicitly identify that
+this is the last tuple in the update chain.
+
+HEAP_LATEST_TUPLE - When this bit is set, the tuple is the last tuple in
+the update chain. The OffsetNumber part of t_ctid points to the root
+line pointer of the chain when HEAP_LATEST_TUPLE flag is set.
+
+If UPDATE operation is aborted, the last tuple in the update chain
+becomes dead. The root line pointer information stored in the tuple
+which remains the last valid tuple in the chain is also lost. In such
+rare cases, the root line pointer must be found in a hard way by
+scanning the entire heap page.
+
+Tracking WARM Chains
+--------------------
+
+When a tuple is WARM updated, the old, the new and every subsequent tuple in
+the chain is marked with a special HEAP_WARM_UPDATED flag. We use the last
+remaining bit in t_infomask2 to store this information.
+
+When a tuple is returned from a WARM chain, the caller must do additional
+checks to ensure that the tuple matches the index key. Even if the tuple
+precedes the WARM update in the chain, it must still be rechecked for the index
+key match (case when old tuple is returned by the new index key). So we must
+follow the update chain everytime to the end to check if this is a WARM
+chain.
+
+Converting WARM chains back to HOT chains (VACUUM ?)
+----------------------------------------------------
+
+The current implementation of WARM allows only one WARM update per
+chain. This simplifies the design and addresses certain issues around
+duplicate key scans. But this also implies that the benefit of WARM will be
+no more than 50%, which is still significant, but if we could return
+WARM chains back to normal status, we could do far more WARM updates.
+
+A distinct property of a WARM chain is that at least one index has more
+than one live index entries pointing to the root of the chain. In other
+words, if we can remove duplicate entry from every index or conclusively
+prove that there are no duplicate index entries for the root line
+pointer, the chain can again be marked as HOT.
+
+Here is one idea:
+
+A WARM chain has two parts, separated by the tuple that caused WARM
+update. All tuples in each part has matching index keys, but certain
+index keys may not match between these two parts. Lets say we mark heap
+tuples in the second part with a special HEAP_WARM_TUPLE flag. Similarly, the
+new index entries caused by the first WARM update are also marked with
+INDEX_WARM_POINTER flags.
+
+There are two distinct parts of the WARM chain. The first part where none of
+the tuples have HEAP_WARM_TUPLE flag set and the second part where every tuple
+has the flag set. Each of these parts satisfy HOT property on its own i.e. all
+tuples have the same value for indexed columns. But these two parts are
+separated by the WARM update which breaks HOT property for one or more indexes.
+
+Heap chain: [1] [2] [3] [4]
+ [aaaa, 1111] -> [aaaa, 1111] -> [bbbb, 1111]W -> [bbbb, 1111]W
+
+Index1: (aaaa) points to 1 (satisfies only tuples without W)
+ (bbbb)W points to 1 (satisfies only tuples marked with W)
+
+Index2: (1111) points to 1 (satisfies tuples with and without W)
+
+
+It's clear that for indexes with both pointers, a heap tuple without
+HEAP_WARM_TUPLE flag will be reachable from the index pointer cleared of
+INDEX_WARM_POINTER flag and that with HEAP_WARM_TUPLE flag will be reachable
+from the pointer with INDEX_WARM_POINTER. But for indexes which did not create
+a new entry, tuples with and without the HEAP_WARM_TUPLE flag will be reachable
+from the original index pointer which doesn't have the INDEX_WARM_POINTER flag.
+(there is no pointer with INDEX_WARM_POINTER in such indexes).
+
+During first heap scan of VACUUM, we look for tuples with HEAP_WARM_UPDATED
+set. If all or none of the live tuples in the chain are marked with
+HEAP_WARM_TUPLE flag, then the chain is a candidate for HOT conversion. We
+remember the root line pointer and whether the tuples in the chain had
+HEAP_WARM_TUPLE flags set or not.
+
+If we have a WARM chain with HEAP_WARM_TUPLE set, then our goal is to remove
+the index pointers without INDEX_WARM_POINTER flags and vice versa. But there
+is a catch. For Index2 above, there is only one pointer and it does not have
+the INDEX_WARM_POINTER flag set. Since all heap tuples are reachable only via
+this pointer, it must not be removed. IOW we should remove index pointer
+without INDEX_WARM_POINTER iff a another index pointer with INDEX_WARM_POINTER
+exists. Since index vacuum may visit these pointers in any order, we will need
+another index pass to detect redundant index pointers, which can safely be
+removed because all live tuples are reachable via the other index pointer. So
+in the first index pass we check which WARM candidates have 2 index pointers.
+In the second pass, we remove the redundant pointer and clear the
+INDEX_WARM_POINTER flag if that's the surviving index pointer. Note that
+all index pointers, either CLEAR or WARM, to dead tuples are removed during the
+first index scan itself.
+
+During the second heap scan, we fix WARM chain by clearing HEAP_WARM_UPDATED
+and HEAP_WARM_TUPLE flags on tuples.
+
+There are some more problems around aborted vacuums. For example, if vacuum
+aborts after clearing INDEX_WARM_POINTER flag but before removing the other
+index pointer, we will end up with two index pointers and none of those will
+have INDEX_WARM_POINTER set. But since the HEAP_WARM_UPDATED flag on the heap
+tuple is still set, further WARM updates to the chain will be blocked. I guess
+we will need some special handling for case with multiple index pointers where
+none of the index pointers has INDEX_WARM_POINTER flag set. We can either leave
+these WARM chains alone and let them die with a subsequent non-WARM update or
+must apply heap-recheck logic during index vacuum to find the dead pointer.
+Given that vacuum-aborts are not common, I am inclined to leave this case
+unhandled. We must still check for presence of multiple index pointers without
+INDEX_WARM_POINTER flags and ensure that we don't accidently remove either of
+these pointers and also must not clear WARM chains.
+
+CREATE INDEX CONCURRENTLY
+-------------------------
+
+Currently CREATE INDEX CONCURRENTLY (CIC) is implemented as a 3-phase
+process. In the first phase, we create catalog entry for the new index
+so that the index is visible to all other backends, but still don't use
+it for either read or write. But we ensure that no new broken HOT
+chains are created by new transactions. In the second phase, we build
+the new index using a MVCC snapshot and then make the index available
+for inserts. We then do another pass over the index and insert any
+missing tuples, everytime indexing only it's root line pointer. See
+README.HOT for details about how HOT impacts CIC and how various
+challenges are tackeled.
+
+WARM poses another challenge because it allows creation of HOT chains
+even when an index key is changed. But since the index is not ready for
+insertion until the second phase is over, we might end up with a
+situation where the HOT chain has tuples with different index columns,
+yet only one of these values are indexed by the new index. Note that
+during the third phase, we only index tuples whose root line pointer is
+missing from the index. But we can't easily check if the existing index
+tuple is actually indexing the heap tuple visible to the new MVCC
+snapshot. Finding that information will require us to query the index
+again for every tuple in the chain, especially if it's a WARM tuple.
+This would require repeated access to the index. Another option would be
+to return index keys along with the heap TIDs when index is scanned for
+collecting all indexed TIDs during third phase. We can then compare the
+heap tuple against the already indexed key and decide whether or not to
+index the new tuple.
+
+We solve this problem more simply by disallowing WARM updates until the
+index is ready for insertion. We don't need to disallow WARM on a
+wholesale basis, but only those updates that change the columns of the
+new index are disallowed to be WARM updates.
diff --git b/src/backend/access/heap/heapam.c a/src/backend/access/heap/heapam.c
index 26a7af4..c86fbc6 100644
--- b/src/backend/access/heap/heapam.c
+++ a/src/backend/access/heap/heapam.c
@@ -1974,6 +1974,206 @@ heap_fetch(Relation relation,
}
/*
+ * Check status of a (possibly) WARM chain.
+ *
+ * This function looks at a HOT/WARM chain starting at tid and return a bitmask
+ * of information. We only follow the chain as long as it's known to be valid
+ * HOT chain. Information returned by the function consists of:
+ *
+ * HCWC_WARM_UPDATED_TUPLE - a tuple with HEAP_WARM_UPDATED is found somewhere
+ * in the chain. Note that when a tuple is WARM
+ * updated, both old and new versions are marked
+ * with this flag/
+ *
+ * HCWC_WARM_TUPLE - a tuple with HEAP_WARM_TUPLE is found somewhere in
+ * the chain.
+ *
+ * HCWC_CLEAR_TUPLE - a tuple without HEAP_WARM_TUPLE is found somewhere in
+ * the chain.
+ *
+ * If stop_at_warm is true, we stop when the first HEAP_WARM_UPDATED tuple is
+ * found and return information collected so far.
+ */
+HeapCheckWarmChainStatus
+heap_check_warm_chain(Page dp, ItemPointer tid, bool stop_at_warm)
+{
+ TransactionId prev_xmax = InvalidTransactionId;
+ OffsetNumber offnum;
+ HeapTupleData heapTuple;
+ HeapCheckWarmChainStatus status = 0;
+
+ offnum = ItemPointerGetOffsetNumber(tid);
+ heapTuple.t_self = *tid;
+ /* Scan through possible multiple members of HOT-chain */
+ for (;;)
+ {
+ ItemId lp;
+
+ /* check for bogus TID */
+ if (offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(dp))
+ break;
+
+ lp = PageGetItemId(dp, offnum);
+
+ /* check for unused, dead, or redirected items */
+ if (!ItemIdIsNormal(lp))
+ {
+ if (ItemIdIsRedirected(lp))
+ {
+ /* Follow the redirect */
+ offnum = ItemIdGetRedirect(lp);
+ continue;
+ }
+ /* else must be end of chain */
+ break;
+ }
+
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(dp, lp);
+ ItemPointerSetOffsetNumber(&heapTuple.t_self, offnum);
+
+ /*
+ * The xmin should match the previous xmax value, else chain is
+ * broken.
+ */
+ if (TransactionIdIsValid(prev_xmax) &&
+ !TransactionIdEquals(prev_xmax,
+ HeapTupleHeaderGetXmin(heapTuple.t_data)))
+ break;
+
+
+ if (HeapTupleHeaderIsWarmUpdated(heapTuple.t_data))
+ {
+ /* We found a WARM_UPDATED tuple */
+ status |= HCWC_WARM_UPDATED_TUPLE;
+
+ /*
+ * If we've been told to stop at the first WARM_UPDATED tuple, just
+ * return whatever information collected so far.
+ */
+ if (stop_at_warm)
+ return status;
+
+ /*
+ * Remember whether it's a CLEAR or a WARM tuple.
+ */
+ if (HeapTupleHeaderIsWarm(heapTuple.t_data))
+ status |= HCWC_WARM_TUPLE;
+ else
+ status |= HCWC_CLEAR_TUPLE;
+ }
+ else
+ /* Must be a regular, non-WARM tuple */
+ status |= HCWC_CLEAR_TUPLE;
+
+ /*
+ * Check to see if HOT chain continues past this tuple; if so fetch
+ * the next offnum and loop around.
+ */
+ if (!HeapTupleIsHotUpdated(&heapTuple))
+ break;
+
+ /*
+ * It can't be a HOT chain if the tuple contains root line pointer
+ */
+ if (HeapTupleHeaderHasRootOffset(heapTuple.t_data))
+ break;
+
+ offnum = ItemPointerGetOffsetNumber(&heapTuple.t_data->t_ctid);
+ prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple.t_data);
+ }
+
+ /* All OK. No need to recheck */
+ return status;
+}
+
+/*
+ * Scan through the WARM chain starting at tid and reset all WARM related
+ * flags. At the end, the chain will have all characteristics of a regular HOT
+ * chain.
+ *
+ * Return the number of cleared offnums. Cleared offnums are returned in the
+ * passed-in cleared_offnums array. The caller must ensure that the array is
+ * large enough to hold maximum offnums that can be cleared by this invokation
+ * of heap_clear_warm_chain().
+ */
+int
+heap_clear_warm_chain(Page dp, ItemPointer tid, OffsetNumber *cleared_offnums)
+{
+ TransactionId prev_xmax = InvalidTransactionId;
+ OffsetNumber offnum;
+ HeapTupleData heapTuple;
+ int num_cleared = 0;
+
+ offnum = ItemPointerGetOffsetNumber(tid);
+ heapTuple.t_self = *tid;
+ /* Scan through possible multiple members of HOT-chain */
+ for (;;)
+ {
+ ItemId lp;
+
+ /* check for bogus TID */
+ if (offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(dp))
+ break;
+
+ lp = PageGetItemId(dp, offnum);
+
+ /* check for unused, dead, or redirected items */
+ if (!ItemIdIsNormal(lp))
+ {
+ if (ItemIdIsRedirected(lp))
+ {
+ /* Follow the redirect */
+ offnum = ItemIdGetRedirect(lp);
+ continue;
+ }
+ /* else must be end of chain */
+ break;
+ }
+
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(dp, lp);
+ ItemPointerSetOffsetNumber(&heapTuple.t_self, offnum);
+
+ /*
+ * The xmin should match the previous xmax value, else chain is
+ * broken.
+ */
+ if (TransactionIdIsValid(prev_xmax) &&
+ !TransactionIdEquals(prev_xmax,
+ HeapTupleHeaderGetXmin(heapTuple.t_data)))
+ break;
+
+
+ /*
+ * Clear WARM_UPDATED and WARM flags.
+ */
+ if (HeapTupleHeaderIsWarmUpdated(heapTuple.t_data))
+ {
+ HeapTupleHeaderClearWarmUpdated(heapTuple.t_data);
+ HeapTupleHeaderClearWarm(heapTuple.t_data);
+ cleared_offnums[num_cleared++] = offnum;
+ }
+
+ /*
+ * Check to see if HOT chain continues past this tuple; if so fetch
+ * the next offnum and loop around.
+ */
+ if (!HeapTupleIsHotUpdated(&heapTuple))
+ break;
+
+ /*
+ * It can't be a HOT chain if the tuple contains root line pointer
+ */
+ if (HeapTupleHeaderHasRootOffset(heapTuple.t_data))
+ break;
+
+ offnum = ItemPointerGetOffsetNumber(&heapTuple.t_data->t_ctid);
+ prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple.t_data);
+ }
+
+ return num_cleared;
+}
+
+/*
* heap_hot_search_buffer - search HOT chain for tuple satisfying snapshot
*
* On entry, *tid is the TID of a tuple (either a simple tuple, or the root
@@ -1993,11 +2193,14 @@ heap_fetch(Relation relation,
* Unlike heap_fetch, the caller must already have pin and (at least) share
* lock on the buffer; it is still pinned/locked at exit. Also unlike
* heap_fetch, we do not report any pgstats count; caller may do so if wanted.
+ *
+ * recheck should be set false on entry by caller, will be set true on exit
+ * if a WARM tuple is encountered.
*/
bool
heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
Snapshot snapshot, HeapTuple heapTuple,
- bool *all_dead, bool first_call)
+ bool *all_dead, bool first_call, bool *recheck)
{
Page dp = (Page) BufferGetPage(buffer);
TransactionId prev_xmax = InvalidTransactionId;
@@ -2051,9 +2254,12 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
ItemPointerSetOffsetNumber(&heapTuple->t_self, offnum);
/*
- * Shouldn't see a HEAP_ONLY tuple at chain start.
+ * Shouldn't see a HEAP_ONLY tuple at chain start, unless we are
+ * dealing with a WARM updated tuple in which case deferred triggers
+ * may request to fetch a WARM tuple from middle of a chain.
*/
- if (at_chain_start && HeapTupleIsHeapOnly(heapTuple))
+ if (at_chain_start && HeapTupleIsHeapOnly(heapTuple) &&
+ !HeapTupleIsWarmUpdated(heapTuple))
break;
/*
@@ -2066,6 +2272,20 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
break;
/*
+ * Check if there exists a WARM tuple somewhere down the chain and set
+ * recheck to TRUE.
+ *
+ * XXX This is not very efficient right now, and we should look for
+ * possible improvements here.
+ */
+ if (recheck && *recheck == false)
+ {
+ HeapCheckWarmChainStatus status;
+ status = heap_check_warm_chain(dp, &heapTuple->t_self, true);
+ *recheck = HCWC_IS_WARM_UPDATED(status);
+ }
+
+ /*
* When first_call is true (and thus, skip is initially false) we'll
* return the first tuple we find. But on later passes, heapTuple
* will initially be pointing to the tuple we returned last time.
@@ -2114,7 +2334,8 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
* Check to see if HOT chain continues past this tuple; if so fetch
* the next offnum and loop around.
*/
- if (HeapTupleIsHotUpdated(heapTuple))
+ if (HeapTupleIsHotUpdated(heapTuple) &&
+ !HeapTupleHeaderHasRootOffset(heapTuple->t_data))
{
Assert(ItemPointerGetBlockNumber(&heapTuple->t_data->t_ctid) ==
ItemPointerGetBlockNumber(tid));
@@ -2138,18 +2359,41 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
*/
bool
heap_hot_search(ItemPointer tid, Relation relation, Snapshot snapshot,
- bool *all_dead)
+ bool *all_dead, bool *recheck, Buffer *cbuffer,
+ HeapTuple heapTuple)
{
bool result;
Buffer buffer;
- HeapTupleData heapTuple;
+ ItemPointerData ret_tid = *tid;
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_SHARE);
- result = heap_hot_search_buffer(tid, relation, buffer, snapshot,
- &heapTuple, all_dead, true);
- LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
- ReleaseBuffer(buffer);
+ result = heap_hot_search_buffer(&ret_tid, relation, buffer, snapshot,
+ heapTuple, all_dead, true, recheck);
+
+ /*
+ * If we are returning a potential candidate tuple from this chain and the
+ * caller has requested for "recheck" hint, keep the buffer locked and
+ * pinned. The caller must release the lock and pin on the buffer in all
+ * such cases.
+ */
+ if (!result || !recheck || !(*recheck))
+ {
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buffer);
+ }
+
+ /*
+ * Set the caller supplied tid with the actual location of the tuple being
+ * returned.
+ */
+ if (result)
+ {
+ *tid = ret_tid;
+ if (cbuffer)
+ *cbuffer = buffer;
+ }
+
return result;
}
@@ -2792,7 +3036,7 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
{
XLogRecPtr recptr;
xl_heap_multi_insert *xlrec;
- uint8 info = XLOG_HEAP2_MULTI_INSERT;
+ uint8 info = XLOG_HEAP_MULTI_INSERT;
char *tupledata;
int totaldatalen;
char *scratchptr = scratch;
@@ -2889,7 +3133,7 @@ heap_multi_insert(Relation relation, HeapTuple *tuples, int ntuples,
/* filtering by origin on a row level is much more efficient */
XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
- recptr = XLogInsert(RM_HEAP2_ID, info);
+ recptr = XLogInsert(RM_HEAP_ID, info);
PageSetLSN(page, recptr);
}
@@ -3313,7 +3557,9 @@ l1:
}
/* store transaction information of xact deleting the tuple */
- tp.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ tp.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(tp.t_data))
+ tp.t_data->t_infomask &= ~HEAP_MOVED;
tp.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
tp.t_data->t_infomask |= new_infomask;
tp.t_data->t_infomask2 |= new_infomask2;
@@ -3508,15 +3754,18 @@ simple_heap_delete(Relation relation, ItemPointer tid)
HTSU_Result
heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
CommandId cid, Snapshot crosscheck, bool wait,
- HeapUpdateFailureData *hufd, LockTupleMode *lockmode)
+ HeapUpdateFailureData *hufd, LockTupleMode *lockmode,
+ Bitmapset **modified_attrsp, bool *warm_update)
{
HTSU_Result result;
TransactionId xid = GetCurrentTransactionId();
Bitmapset *hot_attrs;
Bitmapset *key_attrs;
Bitmapset *id_attrs;
+ Bitmapset *exprindx_attrs;
Bitmapset *interesting_attrs;
Bitmapset *modified_attrs;
+ Bitmapset *notready_attrs;
ItemId lp;
HeapTupleData oldtup;
HeapTuple heaptup;
@@ -3537,6 +3786,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
bool have_tuple_lock = false;
bool iscombo;
bool use_hot_update = false;
+ bool use_warm_update = false;
bool key_intact;
bool all_visible_cleared = false;
bool all_visible_cleared_new = false;
@@ -3561,6 +3811,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
(errcode(ERRCODE_INVALID_TRANSACTION_STATE),
errmsg("cannot update tuples during a parallel operation")));
+ /* Assume no-warm update */
+ if (warm_update)
+ *warm_update = false;
+
/*
* Fetch the list of attributes to be checked for various operations.
*
@@ -3582,10 +3836,17 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);
id_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ exprindx_attrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_EXPR_PREDICATE);
+ notready_attrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_NOTREADY);
+
+
interesting_attrs = bms_add_members(NULL, hot_attrs);
interesting_attrs = bms_add_members(interesting_attrs, key_attrs);
interesting_attrs = bms_add_members(interesting_attrs, id_attrs);
-
+ interesting_attrs = bms_add_members(interesting_attrs, exprindx_attrs);
+ interesting_attrs = bms_add_members(interesting_attrs, notready_attrs);
block = ItemPointerGetBlockNumber(otid);
offnum = ItemPointerGetOffsetNumber(otid);
@@ -3637,6 +3898,9 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
modified_attrs = HeapDetermineModifiedColumns(relation, interesting_attrs,
&oldtup, newtup);
+ if (modified_attrsp)
+ *modified_attrsp = bms_copy(modified_attrs);
+
/*
* If we're not updating any "key" column, we can grab a weaker lock type.
* This allows for more concurrency when we are running simultaneously
@@ -3892,6 +4156,7 @@ l2:
bms_free(hot_attrs);
bms_free(key_attrs);
bms_free(id_attrs);
+ bms_free(exprindx_attrs);
bms_free(modified_attrs);
bms_free(interesting_attrs);
return result;
@@ -4057,7 +4322,9 @@ l2:
START_CRIT_SECTION();
/* Clear obsolete visibility flags ... */
- oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ oldtup.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(oldtup.t_data))
+ oldtup.t_data->t_infomask &= ~HEAP_MOVED;
oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
HeapTupleClearHotUpdated(&oldtup);
/* ... and store info about transaction updating this tuple */
@@ -4210,6 +4477,24 @@ l2:
*/
if (!bms_overlap(modified_attrs, hot_attrs))
use_hot_update = true;
+ else
+ {
+ /*
+ * If no WARM updates yet on this chain, let this update be a WARM
+ * update.
+ *
+ * We check for both warm and warm updated tuples since if the
+ * previous WARM update aborted, we may still have added
+ * another index entry for this HOT chain. In such situations, we
+ * must not attempt a WARM update.
+ */
+ if (relation->rd_supportswarm &&
+ !bms_overlap(modified_attrs, exprindx_attrs) &&
+ !bms_is_subset(hot_attrs, modified_attrs) &&
+ !bms_overlap(notready_attrs, modified_attrs) &&
+ !HeapTupleIsWarmUpdated(&oldtup))
+ use_warm_update = true;
+ }
}
else
{
@@ -4256,6 +4541,32 @@ l2:
HeapTupleSetHeapOnly(heaptup);
/* Mark the caller's copy too, in case different from heaptup */
HeapTupleSetHeapOnly(newtup);
+
+ /*
+ * Even if we are doing a HOT update, we must carry forward the WARM
+ * flag because we may have already inserted another index entry
+ * pointing to our root and a third entry may create duplicates.
+ *
+ * Note: If we ever have a mechanism to avoid duplicate <key, TID> in
+ * indexes, we could look at relaxing this restriction and allow even
+ * more WARM udpates.
+ */
+ if (HeapTupleIsWarmUpdated(&oldtup))
+ {
+ HeapTupleSetWarmUpdated(heaptup);
+ HeapTupleSetWarmUpdated(newtup);
+ }
+
+ /*
+ * If the old tuple is a WARM tuple then mark the new tuple as a WARM
+ * tuple as well.
+ */
+ if (HeapTupleIsWarm(&oldtup))
+ {
+ HeapTupleSetWarm(heaptup);
+ HeapTupleSetWarm(newtup);
+ }
+
/*
* For HOT (or WARM) updated tuples, we store the offset of the root
* line pointer of this chain in the ip_posid field of the new tuple.
@@ -4268,12 +4579,45 @@ l2:
if (HeapTupleHeaderHasRootOffset(oldtup.t_data))
root_offnum = HeapTupleHeaderGetRootOffset(oldtup.t_data);
}
+ else if (use_warm_update)
+ {
+ /* Mark the old tuple as HOT-updated */
+ HeapTupleSetHotUpdated(&oldtup);
+ HeapTupleSetWarmUpdated(&oldtup);
+
+ /* And mark the new tuple as heap-only */
+ HeapTupleSetHeapOnly(heaptup);
+ /* Mark the new tuple as WARM tuple */
+ HeapTupleSetWarmUpdated(heaptup);
+ /* This update also starts the WARM chain */
+ HeapTupleSetWarm(heaptup);
+ Assert(!HeapTupleIsWarm(&oldtup));
+
+ /* Mark the caller's copy too, in case different from heaptup */
+ HeapTupleSetHeapOnly(newtup);
+ HeapTupleSetWarmUpdated(newtup);
+ HeapTupleSetWarm(newtup);
+
+ if (HeapTupleHeaderHasRootOffset(oldtup.t_data))
+ root_offnum = HeapTupleHeaderGetRootOffset(oldtup.t_data);
+ else
+ root_offnum = heap_get_root_tuple(page,
+ ItemPointerGetOffsetNumber(&(oldtup.t_self)));
+
+ /* Let the caller know we did a WARM update */
+ if (warm_update)
+ *warm_update = true;
+ }
else
{
/* Make sure tuples are correctly marked as not-HOT */
HeapTupleClearHotUpdated(&oldtup);
HeapTupleClearHeapOnly(heaptup);
HeapTupleClearHeapOnly(newtup);
+ HeapTupleClearWarmUpdated(heaptup);
+ HeapTupleClearWarmUpdated(newtup);
+ HeapTupleClearWarm(heaptup);
+ HeapTupleClearWarm(newtup);
root_offnum = InvalidOffsetNumber;
}
@@ -4292,7 +4636,9 @@ l2:
HeapTupleHeaderSetHeapLatest(newtup->t_data, root_offnum);
/* Clear obsolete visibility flags, possibly set by ourselves above... */
- oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ oldtup.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(oldtup.t_data))
+ oldtup.t_data->t_infomask &= ~HEAP_MOVED;
oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
/* ... and store info about transaction updating this tuple */
Assert(TransactionIdIsValid(xmax_old_tuple));
@@ -4383,7 +4729,10 @@ l2:
if (have_tuple_lock)
UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode);
- pgstat_count_heap_update(relation, use_hot_update);
+ /*
+ * Count HOT and WARM updates separately
+ */
+ pgstat_count_heap_update(relation, use_hot_update, use_warm_update);
/*
* If heaptup is a private copy, release it. Don't forget to copy t_self
@@ -4523,7 +4872,8 @@ HeapDetermineModifiedColumns(Relation relation, Bitmapset *interesting_cols,
* via ereport().
*/
void
-simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup)
+simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup,
+ Bitmapset **modified_attrs, bool *warm_update)
{
HTSU_Result result;
HeapUpdateFailureData hufd;
@@ -4532,7 +4882,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup)
result = heap_update(relation, otid, tup,
GetCurrentCommandId(true), InvalidSnapshot,
true /* wait for commit */ ,
- &hufd, &lockmode);
+ &hufd, &lockmode, modified_attrs, warm_update);
switch (result)
{
case HeapTupleSelfUpdated:
@@ -6209,7 +6559,9 @@ heap_abort_speculative(Relation relation, HeapTuple tuple)
PageSetPrunable(page, RecentGlobalXmin);
/* store transaction information of xact deleting the tuple */
- tp.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ tp.t_data->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(tp.t_data))
+ tp.t_data->t_infomask &= ~HEAP_MOVED;
tp.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
/*
@@ -6783,7 +7135,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple, TransactionId cutoff_xid,
* Old-style VACUUM FULL is gone, but we have to keep this code as long as
* we support having MOVED_OFF/MOVED_IN tuples in the database.
*/
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
@@ -6802,7 +7154,7 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple, TransactionId cutoff_xid,
* have failed; whereas a non-dead MOVED_IN tuple must mean the
* xvac transaction succeeded.
*/
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
frz->frzflags |= XLH_INVALID_XVAC;
else
frz->frzflags |= XLH_FREEZE_XVAC;
@@ -7272,7 +7624,7 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
return true;
}
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
if (TransactionIdIsNormal(xid))
@@ -7355,7 +7707,7 @@ heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
return true;
}
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
xid = HeapTupleHeaderGetXvac(tuple);
if (TransactionIdIsNormal(xid) &&
@@ -7381,7 +7733,7 @@ HeapTupleHeaderAdvanceLatestRemovedXid(HeapTupleHeader tuple,
TransactionId xmax = HeapTupleHeaderGetUpdateXid(tuple);
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
- if (tuple->t_infomask & HEAP_MOVED)
+ if (HeapTupleHeaderIsMoved(tuple))
{
if (TransactionIdPrecedes(*latestRemovedXid, xvac))
*latestRemovedXid = xvac;
@@ -7430,6 +7782,36 @@ log_heap_cleanup_info(RelFileNode rnode, TransactionId latestRemovedXid)
}
/*
+ * Perform XLogInsert for a heap-warm-clear operation. Caller must already
+ * have modified the buffer and marked it dirty.
+ */
+XLogRecPtr
+log_heap_warmclear(Relation reln, Buffer buffer,
+ OffsetNumber *cleared, int ncleared)
+{
+ xl_heap_warmclear xlrec;
+ XLogRecPtr recptr;
+
+ /* Caller should not call me on a non-WAL-logged relation */
+ Assert(RelationNeedsWAL(reln));
+
+ xlrec.ncleared = ncleared;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfHeapWarmClear);
+
+ XLogRegisterBuffer(0, buffer, REGBUF_STANDARD);
+
+ if (ncleared > 0)
+ XLogRegisterBufData(0, (char *) cleared,
+ ncleared * sizeof(OffsetNumber));
+
+ recptr = XLogInsert(RM_HEAP2_ID, XLOG_HEAP2_WARMCLEAR);
+
+ return recptr;
+}
+
+/*
* Perform XLogInsert for a heap-clean operation. Caller must already
* have modified the buffer and marked it dirty.
*
@@ -7584,6 +7966,7 @@ log_heap_update(Relation reln, Buffer oldbuf,
bool need_tuple_data = RelationIsLogicallyLogged(reln);
bool init;
int bufflags;
+ bool warm_update = false;
/* Caller should not call me on a non-WAL-logged relation */
Assert(RelationNeedsWAL(reln));
@@ -7595,6 +7978,9 @@ log_heap_update(Relation reln, Buffer oldbuf,
else
info = XLOG_HEAP_UPDATE;
+ if (HeapTupleIsWarmUpdated(newtup))
+ warm_update = true;
+
/*
* If the old and new tuple are on the same page, we only need to log the
* parts of the new tuple that were changed. That saves on the amount of
@@ -7668,6 +8054,8 @@ log_heap_update(Relation reln, Buffer oldbuf,
xlrec.flags |= XLH_UPDATE_CONTAINS_OLD_KEY;
}
}
+ if (warm_update)
+ xlrec.flags |= XLH_UPDATE_WARM_UPDATE;
/* If new tuple is the single and first tuple on page... */
if (ItemPointerGetOffsetNumber(&(newtup->t_self)) == FirstOffsetNumber &&
@@ -8082,6 +8470,60 @@ heap_xlog_clean(XLogReaderState *record)
XLogRecordPageWithFreeSpace(rnode, blkno, freespace);
}
+
+/*
+ * Handles HEAP2_WARMCLEAR record type
+ */
+static void
+heap_xlog_warmclear(XLogReaderState *record)
+{
+ XLogRecPtr lsn = record->EndRecPtr;
+ xl_heap_warmclear *xlrec = (xl_heap_warmclear *) XLogRecGetData(record);
+ Buffer buffer;
+ RelFileNode rnode;
+ BlockNumber blkno;
+ XLogRedoAction action;
+
+ XLogRecGetBlockTag(record, 0, &rnode, NULL, &blkno);
+
+ /*
+ * If we have a full-page image, restore it (using a cleanup lock) and
+ * we're done.
+ */
+ action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, true,
+ &buffer);
+ if (action == BLK_NEEDS_REDO)
+ {
+ Page page = (Page) BufferGetPage(buffer);
+ OffsetNumber *cleared;
+ int ncleared;
+ Size datalen;
+ int i;
+
+ cleared = (OffsetNumber *) XLogRecGetBlockData(record, 0, &datalen);
+
+ ncleared = xlrec->ncleared;
+
+ for (i = 0; i < ncleared; i++)
+ {
+ ItemId lp;
+ OffsetNumber offnum = cleared[i];
+ HeapTupleData heapTuple;
+
+ lp = PageGetItemId(page, offnum);
+ heapTuple.t_data = (HeapTupleHeader) PageGetItem(page, lp);
+
+ HeapTupleHeaderClearWarmUpdated(heapTuple.t_data);
+ HeapTupleHeaderClearWarm(heapTuple.t_data);
+ }
+
+ PageSetLSN(page, lsn);
+ MarkBufferDirty(buffer);
+ }
+ if (BufferIsValid(buffer))
+ UnlockReleaseBuffer(buffer);
+}
+
/*
* Replay XLOG_HEAP2_VISIBLE record.
*
@@ -8328,7 +8770,9 @@ heap_xlog_delete(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
HeapTupleHeaderClearHotUpdated(htup);
fix_infomask_from_infobits(xlrec->infobits_set,
@@ -8349,7 +8793,7 @@ heap_xlog_delete(XLogReaderState *record)
if (!HeapTupleHeaderHasRootOffset(htup))
{
OffsetNumber root_offnum;
- root_offnum = heap_get_root_tuple(page, xlrec->offnum);
+ root_offnum = heap_get_root_tuple(page, xlrec->offnum);
HeapTupleHeaderSetHeapLatest(htup, root_offnum);
}
@@ -8645,16 +9089,22 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
Size freespace = 0;
XLogRedoAction oldaction;
XLogRedoAction newaction;
+ bool warm_update = false;
/* initialize to keep the compiler quiet */
oldtup.t_data = NULL;
oldtup.t_len = 0;
+ if (xlrec->flags & XLH_UPDATE_WARM_UPDATE)
+ warm_update = true;
+
XLogRecGetBlockTag(record, 0, &rnode, NULL, &newblk);
if (XLogRecGetBlockTag(record, 1, NULL, NULL, &oldblk))
{
/* HOT updates are never done across pages */
Assert(!hot_update);
+ /* WARM updates are never done across pages */
+ Assert(!warm_update);
}
else
oldblk = newblk;
@@ -8714,6 +9164,11 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
&htup->t_infomask2);
HeapTupleHeaderSetXmax(htup, xlrec->old_xmax);
HeapTupleHeaderSetCmax(htup, FirstCommandId, false);
+
+ /* Mark the old tuple has a WARM tuple */
+ if (warm_update)
+ HeapTupleHeaderSetWarmUpdated(htup);
+
/* Set forward chain link in t_ctid */
HeapTupleHeaderSetNextTid(htup, &newtid);
@@ -8849,6 +9304,10 @@ heap_xlog_update(XLogReaderState *record, bool hot_update)
HeapTupleHeaderSetCmin(htup, FirstCommandId);
HeapTupleHeaderSetXmax(htup, xlrec->new_xmax);
+ /* Mark the new tuple has a WARM tuple */
+ if (warm_update)
+ HeapTupleHeaderSetWarmUpdated(htup);
+
offnum = PageAddItem(page, (Item) htup, newlen, offnum, true, true);
if (offnum == InvalidOffsetNumber)
elog(PANIC, "failed to add tuple");
@@ -8976,7 +9435,9 @@ heap_xlog_lock(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
fix_infomask_from_infobits(xlrec->infobits_set, &htup->t_infomask,
&htup->t_infomask2);
@@ -9055,7 +9516,9 @@ heap_xlog_lock_updated(XLogReaderState *record)
htup = (HeapTupleHeader) PageGetItem(page, lp);
- htup->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
+ htup->t_infomask &= ~HEAP_XMAX_BITS;
+ if (HeapTupleHeaderIsMoved(htup))
+ htup->t_infomask &= ~HEAP_MOVED;
htup->t_infomask2 &= ~HEAP_KEYS_UPDATED;
fix_infomask_from_infobits(xlrec->infobits_set, &htup->t_infomask,
&htup->t_infomask2);
@@ -9124,6 +9587,9 @@ heap_redo(XLogReaderState *record)
case XLOG_HEAP_INSERT:
heap_xlog_insert(record);
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ heap_xlog_multi_insert(record);
+ break;
case XLOG_HEAP_DELETE:
heap_xlog_delete(record);
break;
@@ -9152,7 +9618,7 @@ heap2_redo(XLogReaderState *record)
{
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- switch (info & XLOG_HEAP_OPMASK)
+ switch (info & XLOG_HEAP2_OPMASK)
{
case XLOG_HEAP2_CLEAN:
heap_xlog_clean(record);
@@ -9166,9 +9632,6 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_VISIBLE:
heap_xlog_visible(record);
break;
- case XLOG_HEAP2_MULTI_INSERT:
- heap_xlog_multi_insert(record);
- break;
case XLOG_HEAP2_LOCK_UPDATED:
heap_xlog_lock_updated(record);
break;
@@ -9182,6 +9645,9 @@ heap2_redo(XLogReaderState *record)
case XLOG_HEAP2_REWRITE:
heap_xlog_logical_rewrite(record);
break;
+ case XLOG_HEAP2_WARMCLEAR:
+ heap_xlog_warmclear(record);
+ break;
default:
elog(PANIC, "heap2_redo: unknown op code %u", info);
}
diff --git b/src/backend/access/heap/pruneheap.c a/src/backend/access/heap/pruneheap.c
index f54337c..4e8ed79 100644
--- b/src/backend/access/heap/pruneheap.c
+++ a/src/backend/access/heap/pruneheap.c
@@ -834,6 +834,13 @@ heap_get_root_tuples_internal(Page page, OffsetNumber target_offnum,
if (!HeapTupleHeaderIsHotUpdated(htup))
continue;
+ /*
+ * If the tuple has root line pointer, it must be the end of the
+ * chain
+ */
+ if (HeapTupleHeaderHasRootOffset(htup))
+ break;
+
/* Set up to scan the HOT-chain */
nextoffnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
diff --git b/src/backend/access/heap/rewriteheap.c a/src/backend/access/heap/rewriteheap.c
index 2d3ae9b..bd469ee 100644
--- b/src/backend/access/heap/rewriteheap.c
+++ a/src/backend/access/heap/rewriteheap.c
@@ -404,6 +404,14 @@ rewrite_heap_tuple(RewriteState state,
old_tuple->t_data->t_infomask & HEAP_XACT_MASK;
/*
+ * We must clear the HEAP_WARM_TUPLE flag if the HEAP_WARM_UPDATED is
+ * cleared above.
+ */
+ if (HeapTupleHeaderIsWarmUpdated(old_tuple->t_data))
+ HeapTupleHeaderClearWarm(new_tuple->t_data);
+
+
+ /*
* While we have our hands on the tuple, we may as well freeze any
* eligible xmin or xmax, so that future VACUUM effort can be saved.
*/
@@ -428,7 +436,7 @@ rewrite_heap_tuple(RewriteState state,
memset(&hashkey, 0, sizeof(hashkey));
hashkey.xmin = HeapTupleHeaderGetUpdateXid(old_tuple->t_data);
- /*
+ /*
* We've already checked that this is not the last tuple in the chain,
* so fetch the next TID in the chain.
*/
@@ -737,7 +745,7 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
newitemid = PageGetItemId(page, newoff);
onpage_tup = (HeapTupleHeader) PageGetItem(page, newitemid);
- /*
+ /*
* Set t_ctid just to ensure that block number is copied correctly, but
* then immediately mark the tuple as the latest.
*/
diff --git b/src/backend/access/heap/tuptoaster.c a/src/backend/access/heap/tuptoaster.c
index 19e7048..47b01eb 100644
--- b/src/backend/access/heap/tuptoaster.c
+++ a/src/backend/access/heap/tuptoaster.c
@@ -1620,7 +1620,8 @@ toast_save_datum(Relation rel, Datum value,
toastrel,
toastidxs[i]->rd_index->indisunique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- NULL);
+ NULL,
+ false);
}
/*
diff --git b/src/backend/access/index/genam.c a/src/backend/access/index/genam.c
index a91fda7..d523c8f 100644
--- b/src/backend/access/index/genam.c
+++ a/src/backend/access/index/genam.c
@@ -127,6 +127,8 @@ RelationGetIndexScan(Relation indexRelation, int nkeys, int norderbys)
scan->xs_cbuf = InvalidBuffer;
scan->xs_continue_hot = false;
+ scan->indexInfo = NULL;
+
return scan;
}
diff --git b/src/backend/access/index/indexam.c a/src/backend/access/index/indexam.c
index cc5ac8b..d048714 100644
--- b/src/backend/access/index/indexam.c
+++ a/src/backend/access/index/indexam.c
@@ -197,7 +197,8 @@ index_insert(Relation indexRelation,
ItemPointer heap_t_ctid,
Relation heapRelation,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo,
+ bool warm_update)
{
RELATION_CHECKS;
CHECK_REL_PROCEDURE(aminsert);
@@ -207,6 +208,12 @@ index_insert(Relation indexRelation,
(HeapTuple) NULL,
InvalidBuffer);
+ if (warm_update)
+ {
+ Assert(indexRelation->rd_amroutine->amwarminsert != NULL);
+ return indexRelation->rd_amroutine->amwarminsert(indexRelation, values,
+ isnull, heap_t_ctid, heapRelation, checkUnique, indexInfo);
+ }
return indexRelation->rd_amroutine->aminsert(indexRelation, values, isnull,
heap_t_ctid, heapRelation,
checkUnique, indexInfo);
@@ -291,6 +298,25 @@ index_beginscan_internal(Relation indexRelation,
scan->parallel_scan = pscan;
scan->xs_temp_snap = temp_snap;
+ /*
+ * If the index supports recheck, make sure that index tuple is saved
+ * during index scans. Also build and cache IndexInfo which is used by
+ * amrecheck routine.
+ *
+ * XXX Ideally, we should look at all indexes on the table and check if
+ * WARM is at all supported on the base table. If WARM is not supported
+ * then we don't need to do any recheck. RelationGetIndexAttrBitmap() does
+ * do that and sets rd_supportswarm after looking at all indexes. But we
+ * don't know if the function was called earlier in the session when we're
+ * here. We can't call it now because there exists a risk of causing
+ * deadlock.
+ */
+ if (indexRelation->rd_amroutine->amrecheck)
+ {
+ scan->xs_want_itup = true;
+ scan->indexInfo = BuildIndexInfo(indexRelation);
+ }
+
return scan;
}
@@ -358,6 +384,10 @@ index_endscan(IndexScanDesc scan)
if (scan->xs_temp_snap)
UnregisterSnapshot(scan->xs_snapshot);
+ /* Free cached IndexInfo, if any */
+ if (scan->indexInfo)
+ pfree(scan->indexInfo);
+
/* Release the scan data structure itself */
IndexScanEnd(scan);
}
@@ -535,7 +565,7 @@ index_getnext_tid(IndexScanDesc scan, ScanDirection direction)
/*
* The AM's amgettuple proc finds the next index entry matching the scan
* keys, and puts the TID into scan->xs_ctup.t_self. It should also set
- * scan->xs_recheck and possibly scan->xs_itup/scan->xs_hitup, though we
+ * scan->xs_tuple_recheck and possibly scan->xs_itup/scan->xs_hitup, though we
* pay no attention to those fields here.
*/
found = scan->indexRelation->rd_amroutine->amgettuple(scan, direction);
@@ -574,7 +604,7 @@ index_getnext_tid(IndexScanDesc scan, ScanDirection direction)
* dropped in a future index_getnext_tid, index_fetch_heap or index_endscan
* call).
*
- * Note: caller must check scan->xs_recheck, and perform rechecking of the
+ * Note: caller must check scan->xs_tuple_recheck, and perform rechecking of the
* scan keys if required. We do not do that here because we don't have
* enough information to do it efficiently in the general case.
* ----------------
@@ -585,6 +615,7 @@ index_fetch_heap(IndexScanDesc scan)
ItemPointer tid = &scan->xs_ctup.t_self;
bool all_dead = false;
bool got_heap_tuple;
+ bool tuple_recheck;
/* We can skip the buffer-switching logic if we're in mid-HOT chain. */
if (!scan->xs_continue_hot)
@@ -603,6 +634,8 @@ index_fetch_heap(IndexScanDesc scan)
heap_page_prune_opt(scan->heapRelation, scan->xs_cbuf);
}
+ tuple_recheck = false;
+
/* Obtain share-lock on the buffer so we can examine visibility */
LockBuffer(scan->xs_cbuf, BUFFER_LOCK_SHARE);
got_heap_tuple = heap_hot_search_buffer(tid, scan->heapRelation,
@@ -610,32 +643,60 @@ index_fetch_heap(IndexScanDesc scan)
scan->xs_snapshot,
&scan->xs_ctup,
&all_dead,
- !scan->xs_continue_hot);
- LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
+ !scan->xs_continue_hot,
+ &tuple_recheck);
if (got_heap_tuple)
{
+ bool res = true;
+
+ /*
+ * Ok we got a tuple which satisfies the snapshot, but if its part of a
+ * WARM chain, we must do additional checks to ensure that we are
+ * indeed returning a correct tuple. Note that if the index AM does not
+ * implement amrecheck method, then we don't any additional checks
+ * since WARM must have been disabled on such tables.
+ */
+ if (tuple_recheck && scan->xs_itup &&
+ scan->indexRelation->rd_amroutine->amrecheck)
+ {
+ res = scan->indexRelation->rd_amroutine->amrecheck(
+ scan->indexRelation,
+ scan->indexInfo,
+ scan->xs_itup,
+ scan->heapRelation,
+ &scan->xs_ctup);
+ }
+
+ LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
+
/*
* Only in a non-MVCC snapshot can more than one member of the HOT
* chain be visible.
*/
scan->xs_continue_hot = !IsMVCCSnapshot(scan->xs_snapshot);
pgstat_count_heap_fetch(scan->indexRelation);
- return &scan->xs_ctup;
+
+ if (res)
+ return &scan->xs_ctup;
}
+ else
+ {
+ LockBuffer(scan->xs_cbuf, BUFFER_LOCK_UNLOCK);
- /* We've reached the end of the HOT chain. */
- scan->xs_continue_hot = false;
+ /* We've reached the end of the HOT chain. */
+ scan->xs_continue_hot = false;
- /*
- * If we scanned a whole HOT chain and found only dead tuples, tell index
- * AM to kill its entry for that TID (this will take effect in the next
- * amgettuple call, in index_getnext_tid). We do not do this when in
- * recovery because it may violate MVCC to do so. See comments in
- * RelationGetIndexScan().
- */
- if (!scan->xactStartedInRecovery)
- scan->kill_prior_tuple = all_dead;
+ /*
+ * If we scanned a whole HOT chain and found only dead tuples, tell index
+ * AM to kill its entry for that TID (this will take effect in the next
+ * amgettuple call, in index_getnext_tid). We do not do this when in
+ * recovery because it may violate MVCC to do so. See comments in
+ * RelationGetIndexScan().
+ */
+ if (!scan->xactStartedInRecovery)
+ scan->kill_prior_tuple = all_dead;
+ }
return NULL;
}
diff --git b/src/backend/access/nbtree/nbtinsert.c a/src/backend/access/nbtree/nbtinsert.c
index 6dca810..463d4bf 100644
--- b/src/backend/access/nbtree/nbtinsert.c
+++ a/src/backend/access/nbtree/nbtinsert.c
@@ -20,6 +20,7 @@
#include "access/nbtxlog.h"
#include "access/transam.h"
#include "access/xloginsert.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
@@ -250,6 +251,10 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
BTPageOpaque opaque;
Buffer nbuf = InvalidBuffer;
bool found = false;
+ Buffer buffer;
+ HeapTupleData heapTuple;
+ bool recheck = false;
+ IndexInfo *indexInfo = BuildIndexInfo(rel);
/* Assume unique until we find a duplicate */
*is_unique = true;
@@ -309,6 +314,8 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
curitup = (IndexTuple) PageGetItem(page, curitemid);
htid = curitup->t_tid;
+ recheck = false;
+
/*
* If we are doing a recheck, we expect to find the tuple we
* are rechecking. It's not a duplicate, but we have to keep
@@ -326,112 +333,153 @@ _bt_check_unique(Relation rel, IndexTuple itup, Relation heapRel,
* have just a single index entry for the entire chain.
*/
else if (heap_hot_search(&htid, heapRel, &SnapshotDirty,
- &all_dead))
+ &all_dead, &recheck, &buffer,
+ &heapTuple))
{
TransactionId xwait;
+ bool result = true;
/*
- * It is a duplicate. If we are only doing a partial
- * check, then don't bother checking if the tuple is being
- * updated in another transaction. Just return the fact
- * that it is a potential conflict and leave the full
- * check till later.
+ * If the tuple was WARM update, we may again see our own
+ * tuple. Since WARM updates don't create new index
+ * entries, our own tuple is only reachable via the old
+ * index pointer.
*/
- if (checkUnique == UNIQUE_CHECK_PARTIAL)
+ if (checkUnique == UNIQUE_CHECK_EXISTING &&
+ ItemPointerCompare(&htid, &itup->t_tid) == 0)
{
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- *is_unique = false;
- return InvalidTransactionId;
+ found = true;
+ result = false;
+ if (recheck)
+ UnlockReleaseBuffer(buffer);
}
-
- /*
- * If this tuple is being updated by other transaction
- * then we have to wait for its commit/abort.
- */
- xwait = (TransactionIdIsValid(SnapshotDirty.xmin)) ?
- SnapshotDirty.xmin : SnapshotDirty.xmax;
-
- if (TransactionIdIsValid(xwait))
+ else if (recheck)
{
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- /* Tell _bt_doinsert to wait... */
- *speculativeToken = SnapshotDirty.speculativeToken;
- return xwait;
+ result = btrecheck(rel, indexInfo, curitup, heapRel, &heapTuple);
+ UnlockReleaseBuffer(buffer);
}
- /*
- * Otherwise we have a definite conflict. But before
- * complaining, look to see if the tuple we want to insert
- * is itself now committed dead --- if so, don't complain.
- * This is a waste of time in normal scenarios but we must
- * do it to support CREATE INDEX CONCURRENTLY.
- *
- * We must follow HOT-chains here because during
- * concurrent index build, we insert the root TID though
- * the actual tuple may be somewhere in the HOT-chain.
- * While following the chain we might not stop at the
- * exact tuple which triggered the insert, but that's OK
- * because if we find a live tuple anywhere in this chain,
- * we have a unique key conflict. The other live tuple is
- * not part of this chain because it had a different index
- * entry.
- */
- htid = itup->t_tid;
- if (heap_hot_search(&htid, heapRel, SnapshotSelf, NULL))
- {
- /* Normal case --- it's still live */
- }
- else
+ if (result)
{
/*
- * It's been deleted, so no error, and no need to
- * continue searching
+ * It is a duplicate. If we are only doing a partial
+ * check, then don't bother checking if the tuple is being
+ * updated in another transaction. Just return the fact
+ * that it is a potential conflict and leave the full
+ * check till later.
*/
- break;
- }
+ if (checkUnique == UNIQUE_CHECK_PARTIAL)
+ {
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ *is_unique = false;
+ return InvalidTransactionId;
+ }
- /*
- * Check for a conflict-in as we would if we were going to
- * write to this page. We aren't actually going to write,
- * but we want a chance to report SSI conflicts that would
- * otherwise be masked by this unique constraint
- * violation.
- */
- CheckForSerializableConflictIn(rel, NULL, buf);
+ /*
+ * If this tuple is being updated by other transaction
+ * then we have to wait for its commit/abort.
+ */
+ xwait = (TransactionIdIsValid(SnapshotDirty.xmin)) ?
+ SnapshotDirty.xmin : SnapshotDirty.xmax;
+
+ if (TransactionIdIsValid(xwait))
+ {
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ /* Tell _bt_doinsert to wait... */
+ *speculativeToken = SnapshotDirty.speculativeToken;
+ return xwait;
+ }
- /*
- * This is a definite conflict. Break the tuple down into
- * datums and report the error. But first, make sure we
- * release the buffer locks we're holding ---
- * BuildIndexValueDescription could make catalog accesses,
- * which in the worst case might touch this same index and
- * cause deadlocks.
- */
- if (nbuf != InvalidBuffer)
- _bt_relbuf(rel, nbuf);
- _bt_relbuf(rel, buf);
+ /*
+ * Otherwise we have a definite conflict. But before
+ * complaining, look to see if the tuple we want to insert
+ * is itself now committed dead --- if so, don't complain.
+ * This is a waste of time in normal scenarios but we must
+ * do it to support CREATE INDEX CONCURRENTLY.
+ *
+ * We must follow HOT-chains here because during
+ * concurrent index build, we insert the root TID though
+ * the actual tuple may be somewhere in the HOT-chain.
+ * While following the chain we might not stop at the
+ * exact tuple which triggered the insert, but that's OK
+ * because if we find a live tuple anywhere in this chain,
+ * we have a unique key conflict. The other live tuple is
+ * not part of this chain because it had a different index
+ * entry.
+ */
+ recheck = false;
+ ItemPointerCopy(&itup->t_tid, &htid);
+ if (heap_hot_search(&htid, heapRel, SnapshotSelf, NULL,
+ &recheck, &buffer, &heapTuple))
+ {
+ bool result = true;
+ if (recheck)
+ {
+ /*
+ * Recheck if the tuple actually satisfies the
+ * index key. Otherwise, we might be following
+ * a wrong index pointer and mustn't entertain
+ * this tuple.
+ */
+ result = btrecheck(rel, indexInfo, itup, heapRel, &heapTuple);
+ UnlockReleaseBuffer(buffer);
+ }
+ if (!result)
+ break;
+ /* Normal case --- it's still live */
+ }
+ else
+ {
+ /*
+ * It's been deleted, so no error, and no need to
+ * continue searching.
+ */
+ break;
+ }
- {
- Datum values[INDEX_MAX_KEYS];
- bool isnull[INDEX_MAX_KEYS];
- char *key_desc;
-
- index_deform_tuple(itup, RelationGetDescr(rel),
- values, isnull);
-
- key_desc = BuildIndexValueDescription(rel, values,
- isnull);
-
- ereport(ERROR,
- (errcode(ERRCODE_UNIQUE_VIOLATION),
- errmsg("duplicate key value violates unique constraint \"%s\"",
- RelationGetRelationName(rel)),
- key_desc ? errdetail("Key %s already exists.",
- key_desc) : 0,
- errtableconstraint(heapRel,
- RelationGetRelationName(rel))));
+ /*
+ * Check for a conflict-in as we would if we were going to
+ * write to this page. We aren't actually going to write,
+ * but we want a chance to report SSI conflicts that would
+ * otherwise be masked by this unique constraint
+ * violation.
+ */
+ CheckForSerializableConflictIn(rel, NULL, buf);
+
+ /*
+ * This is a definite conflict. Break the tuple down into
+ * datums and report the error. But first, make sure we
+ * release the buffer locks we're holding ---
+ * BuildIndexValueDescription could make catalog accesses,
+ * which in the worst case might touch this same index and
+ * cause deadlocks.
+ */
+ if (nbuf != InvalidBuffer)
+ _bt_relbuf(rel, nbuf);
+ _bt_relbuf(rel, buf);
+
+ {
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ char *key_desc;
+
+ index_deform_tuple(itup, RelationGetDescr(rel),
+ values, isnull);
+
+ key_desc = BuildIndexValueDescription(rel, values,
+ isnull);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_UNIQUE_VIOLATION),
+ errmsg("duplicate key value violates unique constraint \"%s\"",
+ RelationGetRelationName(rel)),
+ key_desc ? errdetail("Key %s already exists.",
+ key_desc) : 0,
+ errtableconstraint(heapRel,
+ RelationGetRelationName(rel))));
+ }
}
}
else if (all_dead)
diff --git b/src/backend/access/nbtree/nbtpage.c a/src/backend/access/nbtree/nbtpage.c
index f815fd4..061c8d4 100644
--- b/src/backend/access/nbtree/nbtpage.c
+++ a/src/backend/access/nbtree/nbtpage.c
@@ -766,29 +766,20 @@ _bt_page_recyclable(Page page)
}
/*
- * Delete item(s) from a btree page during VACUUM.
+ * Delete item(s) and clear WARM item(s) on a btree page during VACUUM.
*
* This must only be used for deleting leaf items. Deleting an item on a
* non-leaf page has to be done as part of an atomic action that includes
- * deleting the page it points to.
+ * deleting the page it points to. We don't ever clear pointers on a non-leaf
+ * page.
*
* This routine assumes that the caller has pinned and locked the buffer.
* Also, the given itemnos *must* appear in increasing order in the array.
- *
- * We record VACUUMs and b-tree deletes differently in WAL. InHotStandby
- * we need to be able to pin all of the blocks in the btree in physical
- * order when replaying the effects of a VACUUM, just as we do for the
- * original VACUUM itself. lastBlockVacuumed allows us to tell whether an
- * intermediate range of blocks has had no changes at all by VACUUM,
- * and so must be scanned anyway during replay. We always write a WAL record
- * for the last block in the index, whether or not it contained any items
- * to be removed. This allows us to scan right up to end of index to
- * ensure correct locking.
*/
void
-_bt_delitems_vacuum(Relation rel, Buffer buf,
- OffsetNumber *itemnos, int nitems,
- BlockNumber lastBlockVacuumed)
+_bt_handleitems_vacuum(Relation rel, Buffer buf,
+ OffsetNumber *delitemnos, int ndelitems,
+ OffsetNumber *clearitemnos, int nclearitems)
{
Page page = BufferGetPage(buf);
BTPageOpaque opaque;
@@ -796,9 +787,20 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
/* No ereport(ERROR) until changes are logged */
START_CRIT_SECTION();
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the array and
+ * hence offnums recorded earlier won't make any sense after
+ * PageIndexMultiDelete is called.
+ */
+ if (nclearitems > 0)
+ _bt_clear_items(page, clearitemnos, nclearitems);
+
/* Fix the page */
- if (nitems > 0)
- PageIndexMultiDelete(page, itemnos, nitems);
+ if (ndelitems > 0)
+ PageIndexMultiDelete(page, delitemnos, ndelitems);
/*
* We can clear the vacuum cycle ID since this page has certainly been
@@ -824,7 +826,8 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
XLogRecPtr recptr;
xl_btree_vacuum xlrec_vacuum;
- xlrec_vacuum.lastBlockVacuumed = lastBlockVacuumed;
+ xlrec_vacuum.ndelitems = ndelitems;
+ xlrec_vacuum.nclearitems = nclearitems;
XLogBeginInsert();
XLogRegisterBuffer(0, buf, REGBUF_STANDARD);
@@ -835,8 +838,11 @@ _bt_delitems_vacuum(Relation rel, Buffer buf,
* is. When XLogInsert stores the whole buffer, the offsets array
* need not be stored too.
*/
- if (nitems > 0)
- XLogRegisterBufData(0, (char *) itemnos, nitems * sizeof(OffsetNumber));
+ if (ndelitems > 0)
+ XLogRegisterBufData(0, (char *) delitemnos, ndelitems * sizeof(OffsetNumber));
+
+ if (nclearitems > 0)
+ XLogRegisterBufData(0, (char *) clearitemnos, nclearitems * sizeof(OffsetNumber));
recptr = XLogInsert(RM_BTREE_ID, XLOG_BTREE_VACUUM);
@@ -1882,3 +1888,13 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, bool *rightsib_empty)
return true;
}
+
+/*
+ * Currently just a wrapper arounf PageIndexClearWarmTuples, but in theory each
+ * index may have it's own way to handle WARM tuples.
+ */
+void
+_bt_clear_items(Page page, OffsetNumber *clearitemnos, uint16 nclearitems)
+{
+ PageIndexClearWarmTuples(page, clearitemnos, nclearitems);
+}
diff --git b/src/backend/access/nbtree/nbtree.c a/src/backend/access/nbtree/nbtree.c
index 775f2ff..6d558af 100644
--- b/src/backend/access/nbtree/nbtree.c
+++ a/src/backend/access/nbtree/nbtree.c
@@ -146,6 +146,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->ambuild = btbuild;
amroutine->ambuildempty = btbuildempty;
amroutine->aminsert = btinsert;
+ amroutine->amwarminsert = btwarminsert;
amroutine->ambulkdelete = btbulkdelete;
amroutine->amvacuumcleanup = btvacuumcleanup;
amroutine->amcanreturn = btcanreturn;
@@ -163,6 +164,7 @@ bthandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = btestimateparallelscan;
amroutine->aminitparallelscan = btinitparallelscan;
amroutine->amparallelrescan = btparallelrescan;
+ amroutine->amrecheck = btrecheck;
PG_RETURN_POINTER(amroutine);
}
@@ -315,11 +317,12 @@ btbuildempty(Relation index)
* Descend the tree recursively, find the appropriate location for our
* new tuple, and put it there.
*/
-bool
-btinsert(Relation rel, Datum *values, bool *isnull,
+static bool
+btinsert_internal(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
- IndexInfo *indexInfo)
+ IndexInfo *indexInfo,
+ bool warm_update)
{
bool result;
IndexTuple itup;
@@ -328,6 +331,11 @@ btinsert(Relation rel, Datum *values, bool *isnull,
itup = index_form_tuple(RelationGetDescr(rel), values, isnull);
itup->t_tid = *ht_ctid;
+ if (warm_update)
+ ItemPointerSetFlags(&itup->t_tid, BTREE_INDEX_WARM_POINTER);
+ else
+ ItemPointerClearFlags(&itup->t_tid);
+
result = _bt_doinsert(rel, itup, checkUnique, heapRel);
pfree(itup);
@@ -335,6 +343,26 @@ btinsert(Relation rel, Datum *values, bool *isnull,
return result;
}
+bool
+btinsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return btinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, false);
+}
+
+bool
+btwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ IndexInfo *indexInfo)
+{
+ return btinsert_internal(rel, values, isnull, ht_ctid, heapRel,
+ checkUnique, indexInfo, true);
+}
+
/*
* btgettuple() -- Get the next tuple in the scan.
*/
@@ -1103,7 +1131,7 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
RBM_NORMAL, info->strategy);
LockBufferForCleanup(buf);
_bt_checkpage(rel, buf);
- _bt_delitems_vacuum(rel, buf, NULL, 0, vstate.lastBlockVacuumed);
+ _bt_handleitems_vacuum(rel, buf, NULL, 0, NULL, 0);
_bt_relbuf(rel, buf);
}
@@ -1201,6 +1229,8 @@ restart:
{
OffsetNumber deletable[MaxOffsetNumber];
int ndeletable;
+ OffsetNumber clearwarm[MaxOffsetNumber];
+ int nclearwarm;
OffsetNumber offnum,
minoff,
maxoff;
@@ -1239,7 +1269,7 @@ restart:
* Scan over all items to see which ones need deleted according to the
* callback function.
*/
- ndeletable = 0;
+ ndeletable = nclearwarm = 0;
minoff = P_FIRSTDATAKEY(opaque);
maxoff = PageGetMaxOffsetNumber(page);
if (callback)
@@ -1250,6 +1280,9 @@ restart:
{
IndexTuple itup;
ItemPointer htup;
+ int flags;
+ bool is_warm = false;
+ IndexBulkDeleteCallbackResult result;
itup = (IndexTuple) PageGetItem(page,
PageGetItemId(page, offnum));
@@ -1276,16 +1309,36 @@ restart:
* applies to *any* type of index that marks index tuples as
* killed.
*/
- if (callback(htup, callback_state))
+ flags = ItemPointerGetFlags(&itup->t_tid);
+ is_warm = ((flags & BTREE_INDEX_WARM_POINTER) != 0);
+
+ if (is_warm)
+ stats->num_warm_pointers++;
+ else
+ stats->num_clear_pointers++;
+
+ result = callback(htup, is_warm, callback_state);
+ if (result == IBDCR_DELETE)
+ {
+ if (is_warm)
+ stats->warm_pointers_removed++;
+ else
+ stats->clear_pointers_removed++;
deletable[ndeletable++] = offnum;
+ }
+ else if (result == IBDCR_CLEAR_WARM)
+ {
+ clearwarm[nclearwarm++] = offnum;
+ }
}
}
/*
- * Apply any needed deletes. We issue just one _bt_delitems_vacuum()
- * call per page, so as to minimize WAL traffic.
+ * Apply any needed deletes and clearing. We issue just one
+ * _bt_handleitems_vacuum() call per page, so as to minimize WAL
+ * traffic.
*/
- if (ndeletable > 0)
+ if (ndeletable > 0 || nclearwarm > 0)
{
/*
* Notice that the issued XLOG_BTREE_VACUUM WAL record includes
@@ -1301,8 +1354,8 @@ restart:
* doesn't seem worth the amount of bookkeeping it'd take to avoid
* that.
*/
- _bt_delitems_vacuum(rel, buf, deletable, ndeletable,
- vstate->lastBlockVacuumed);
+ _bt_handleitems_vacuum(rel, buf, deletable, ndeletable,
+ clearwarm, nclearwarm);
/*
* Remember highest leaf page number we've issued a
@@ -1312,6 +1365,7 @@ restart:
vstate->lastBlockVacuumed = blkno;
stats->tuples_removed += ndeletable;
+ stats->pointers_cleared += nclearwarm;
/* must recompute maxoff */
maxoff = PageGetMaxOffsetNumber(page);
}
diff --git b/src/backend/access/nbtree/nbtutils.c a/src/backend/access/nbtree/nbtutils.c
index 5b259a3..8dab5a8 100644
--- b/src/backend/access/nbtree/nbtutils.c
+++ a/src/backend/access/nbtree/nbtutils.c
@@ -20,11 +20,13 @@
#include "access/nbtree.h"
#include "access/reloptions.h"
#include "access/relscan.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "utils/array.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/datum.h"
typedef struct BTSortArrayContext
@@ -2069,3 +2071,64 @@ btproperty(Oid index_oid, int attno,
return false; /* punt to generic code */
}
}
+
+/*
+ * Check if the index tuple's key matches the one computed from the given heap
+ * tuple's attribute
+ */
+bool
+btrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple)
+{
+ Datum values[INDEX_MAX_KEYS];
+ bool isnull[INDEX_MAX_KEYS];
+ bool isavail[INDEX_MAX_KEYS];
+ int i;
+ bool equal;
+ int natts = indexRel->rd_rel->relnatts;
+ Form_pg_attribute att;
+
+ FormIndexPlainDatum(indexInfo, heapRel, heapTuple, values, isnull, isavail);
+
+ equal = true;
+ for (i = 1; i <= natts; i++)
+ {
+ Datum indxvalue;
+ bool indxisnull;
+
+ /* No need to compare if the attribute value is not available */
+ if (!isavail[i - 1])
+ continue;
+
+ indxvalue = index_getattr(indexTuple, i, indexRel->rd_att, &indxisnull);
+
+ /*
+ * If both are NULL, then they are equal
+ */
+ if (isnull[i - 1] && indxisnull)
+ continue;
+
+ /*
+ * If just one is NULL, then they are not equal
+ */
+ if (isnull[i - 1] || indxisnull)
+ {
+ equal = false;
+ break;
+ }
+
+ /*
+ * Now just do a raw memory comparison. If the index tuple was formed
+ * using this heap tuple, the computed index values must match
+ */
+ att = indexRel->rd_att->attrs[i - 1];
+ if (!datumIsEqual(values[i - 1], indxvalue, att->attbyval,
+ att->attlen))
+ {
+ equal = false;
+ break;
+ }
+ }
+
+ return equal;
+}
diff --git b/src/backend/access/nbtree/nbtxlog.c a/src/backend/access/nbtree/nbtxlog.c
index ac60db0..92be5c8 100644
--- b/src/backend/access/nbtree/nbtxlog.c
+++ a/src/backend/access/nbtree/nbtxlog.c
@@ -390,83 +390,9 @@ btree_xlog_vacuum(XLogReaderState *record)
Buffer buffer;
Page page;
BTPageOpaque opaque;
-#ifdef UNUSED
xl_btree_vacuum *xlrec = (xl_btree_vacuum *) XLogRecGetData(record);
/*
- * This section of code is thought to be no longer needed, after analysis
- * of the calling paths. It is retained to allow the code to be reinstated
- * if a flaw is revealed in that thinking.
- *
- * If we are running non-MVCC scans using this index we need to do some
- * additional work to ensure correctness, which is known as a "pin scan"
- * described in more detail in next paragraphs. We used to do the extra
- * work in all cases, whereas we now avoid that work in most cases. If
- * lastBlockVacuumed is set to InvalidBlockNumber then we skip the
- * additional work required for the pin scan.
- *
- * Avoiding this extra work is important since it requires us to touch
- * every page in the index, so is an O(N) operation. Worse, it is an
- * operation performed in the foreground during redo, so it delays
- * replication directly.
- *
- * If queries might be active then we need to ensure every leaf page is
- * unpinned between the lastBlockVacuumed and the current block, if there
- * are any. This prevents replay of the VACUUM from reaching the stage of
- * removing heap tuples while there could still be indexscans "in flight"
- * to those particular tuples for those scans which could be confused by
- * finding new tuples at the old TID locations (see nbtree/README).
- *
- * It might be worth checking if there are actually any backends running;
- * if not, we could just skip this.
- *
- * Since VACUUM can visit leaf pages out-of-order, it might issue records
- * with lastBlockVacuumed >= block; that's not an error, it just means
- * nothing to do now.
- *
- * Note: since we touch all pages in the range, we will lock non-leaf
- * pages, and also any empty (all-zero) pages that may be in the index. It
- * doesn't seem worth the complexity to avoid that. But it's important
- * that HotStandbyActiveInReplay() will not return true if the database
- * isn't yet consistent; so we need not fear reading still-corrupt blocks
- * here during crash recovery.
- */
- if (HotStandbyActiveInReplay() && BlockNumberIsValid(xlrec->lastBlockVacuumed))
- {
- RelFileNode thisrnode;
- BlockNumber thisblkno;
- BlockNumber blkno;
-
- XLogRecGetBlockTag(record, 0, &thisrnode, NULL, &thisblkno);
-
- for (blkno = xlrec->lastBlockVacuumed + 1; blkno < thisblkno; blkno++)
- {
- /*
- * We use RBM_NORMAL_NO_LOG mode because it's not an error
- * condition to see all-zero pages. The original btvacuumpage
- * scan would have skipped over all-zero pages, noting them in FSM
- * but not bothering to initialize them just yet; so we mustn't
- * throw an error here. (We could skip acquiring the cleanup lock
- * if PageIsNew, but it's probably not worth the cycles to test.)
- *
- * XXX we don't actually need to read the block, we just need to
- * confirm it is unpinned. If we had a special call into the
- * buffer manager we could optimise this so that if the block is
- * not in shared_buffers we confirm it as unpinned. Optimizing
- * this is now moot, since in most cases we avoid the scan.
- */
- buffer = XLogReadBufferExtended(thisrnode, MAIN_FORKNUM, blkno,
- RBM_NORMAL_NO_LOG);
- if (BufferIsValid(buffer))
- {
- LockBufferForCleanup(buffer);
- UnlockReleaseBuffer(buffer);
- }
- }
- }
-#endif
-
- /*
* Like in btvacuumpage(), we need to take a cleanup lock on every leaf
* page. See nbtree/README for details.
*/
@@ -482,19 +408,30 @@ btree_xlog_vacuum(XLogReaderState *record)
if (len > 0)
{
- OffsetNumber *unused;
- OffsetNumber *unend;
+ OffsetNumber *offnums = (OffsetNumber *) ptr;
- unused = (OffsetNumber *) ptr;
- unend = (OffsetNumber *) ((char *) ptr + len);
+ /*
+ * Clear the WARM pointers.
+ *
+ * We must do this before dealing with the dead items because
+ * PageIndexMultiDelete may move items around to compactify the
+ * array and hence offnums recorded earlier won't make any sense
+ * after PageIndexMultiDelete is called.
+ */
+ if (xlrec->nclearitems > 0)
+ _bt_clear_items(page, offnums + xlrec->ndelitems,
+ xlrec->nclearitems);
- if ((unend - unused) > 0)
- PageIndexMultiDelete(page, unused, unend - unused);
+ /*
+ * And handle the deleted items too
+ */
+ if (xlrec->ndelitems > 0)
+ PageIndexMultiDelete(page, offnums, xlrec->ndelitems);
}
/*
* Mark the page as not containing any LP_DEAD items --- see comments
- * in _bt_delitems_vacuum().
+ * in _bt_handleitems_vacuum().
*/
opaque = (BTPageOpaque) PageGetSpecialPointer(page);
opaque->btpo_flags &= ~BTP_HAS_GARBAGE;
diff --git b/src/backend/access/rmgrdesc/heapdesc.c a/src/backend/access/rmgrdesc/heapdesc.c
index 44d2d63..d373e61 100644
--- b/src/backend/access/rmgrdesc/heapdesc.c
+++ a/src/backend/access/rmgrdesc/heapdesc.c
@@ -44,6 +44,12 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
+ else if (info == XLOG_HEAP_MULTI_INSERT)
+ {
+ xl_heap_multi_insert *xlrec = (xl_heap_multi_insert *) rec;
+
+ appendStringInfo(buf, "%d tuples", xlrec->ntuples);
+ }
else if (info == XLOG_HEAP_DELETE)
{
xl_heap_delete *xlrec = (xl_heap_delete *) rec;
@@ -102,7 +108,7 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- info &= XLOG_HEAP_OPMASK;
+ info &= XLOG_HEAP2_OPMASK;
if (info == XLOG_HEAP2_CLEAN)
{
xl_heap_clean *xlrec = (xl_heap_clean *) rec;
@@ -129,12 +135,6 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "cutoff xid %u flags %d",
xlrec->cutoff_xid, xlrec->flags);
}
- else if (info == XLOG_HEAP2_MULTI_INSERT)
- {
- xl_heap_multi_insert *xlrec = (xl_heap_multi_insert *) rec;
-
- appendStringInfo(buf, "%d tuples", xlrec->ntuples);
- }
else if (info == XLOG_HEAP2_LOCK_UPDATED)
{
xl_heap_lock_updated *xlrec = (xl_heap_lock_updated *) rec;
@@ -171,6 +171,12 @@ heap_identify(uint8 info)
case XLOG_HEAP_INSERT | XLOG_HEAP_INIT_PAGE:
id = "INSERT+INIT";
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ id = "MULTI_INSERT";
+ break;
+ case XLOG_HEAP_MULTI_INSERT | XLOG_HEAP_INIT_PAGE:
+ id = "MULTI_INSERT+INIT";
+ break;
case XLOG_HEAP_DELETE:
id = "DELETE";
break;
@@ -219,12 +225,6 @@ heap2_identify(uint8 info)
case XLOG_HEAP2_VISIBLE:
id = "VISIBLE";
break;
- case XLOG_HEAP2_MULTI_INSERT:
- id = "MULTI_INSERT";
- break;
- case XLOG_HEAP2_MULTI_INSERT | XLOG_HEAP_INIT_PAGE:
- id = "MULTI_INSERT+INIT";
- break;
case XLOG_HEAP2_LOCK_UPDATED:
id = "LOCK_UPDATED";
break;
diff --git b/src/backend/access/rmgrdesc/nbtdesc.c a/src/backend/access/rmgrdesc/nbtdesc.c
index fbde9d6..6b2c5d6 100644
--- b/src/backend/access/rmgrdesc/nbtdesc.c
+++ a/src/backend/access/rmgrdesc/nbtdesc.c
@@ -48,8 +48,8 @@ btree_desc(StringInfo buf, XLogReaderState *record)
{
xl_btree_vacuum *xlrec = (xl_btree_vacuum *) rec;
- appendStringInfo(buf, "lastBlockVacuumed %u",
- xlrec->lastBlockVacuumed);
+ appendStringInfo(buf, "ndelitems %u, nclearitems %u",
+ xlrec->ndelitems, xlrec->nclearitems);
break;
}
case XLOG_BTREE_DELETE:
diff --git b/src/backend/access/spgist/spgutils.c a/src/backend/access/spgist/spgutils.c
index e57ac49..59ef7f3 100644
--- b/src/backend/access/spgist/spgutils.c
+++ a/src/backend/access/spgist/spgutils.c
@@ -72,6 +72,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->amestimateparallelscan = NULL;
amroutine->aminitparallelscan = NULL;
amroutine->amparallelrescan = NULL;
+ amroutine->amrecheck = NULL;
PG_RETURN_POINTER(amroutine);
}
diff --git b/src/backend/access/spgist/spgvacuum.c a/src/backend/access/spgist/spgvacuum.c
index cce9b3f..711d351 100644
--- b/src/backend/access/spgist/spgvacuum.c
+++ a/src/backend/access/spgist/spgvacuum.c
@@ -155,7 +155,8 @@ vacuumLeafPage(spgBulkDeleteState *bds, Relation index, Buffer buffer,
{
Assert(ItemPointerIsValid(<->heapPtr));
- if (bds->callback(<->heapPtr, bds->callback_state))
+ if (bds->callback(<->heapPtr, false, bds->callback_state) ==
+ IBDCR_DELETE)
{
bds->stats->tuples_removed += 1;
deletable[i] = true;
@@ -425,7 +426,8 @@ vacuumLeafRoot(spgBulkDeleteState *bds, Relation index, Buffer buffer)
{
Assert(ItemPointerIsValid(<->heapPtr));
- if (bds->callback(<->heapPtr, bds->callback_state))
+ if (bds->callback(<->heapPtr, false, bds->callback_state) ==
+ IBDCR_DELETE)
{
bds->stats->tuples_removed += 1;
toDelete[xlrec.nDelete] = i;
@@ -902,10 +904,10 @@ spgbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
}
/* Dummy callback to delete no tuples during spgvacuumcleanup */
-static bool
-dummy_callback(ItemPointer itemptr, void *state)
+static IndexBulkDeleteCallbackResult
+dummy_callback(ItemPointer itemptr, bool is_warm, void *state)
{
- return false;
+ return IBDCR_KEEP;
}
/*
diff --git b/src/backend/catalog/index.c a/src/backend/catalog/index.c
index 8d42a34..67e68d1 100644
--- b/src/backend/catalog/index.c
+++ a/src/backend/catalog/index.c
@@ -54,6 +54,7 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "optimizer/clauses.h"
+#include "optimizer/var.h"
#include "parser/parser.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
@@ -114,7 +115,7 @@ static void IndexCheckExclusion(Relation heapRelation,
IndexInfo *indexInfo);
static inline int64 itemptr_encode(ItemPointer itemptr);
static inline void itemptr_decode(ItemPointer itemptr, int64 encoded);
-static bool validate_index_callback(ItemPointer itemptr, void *opaque);
+static IndexBulkDeleteCallbackResult validate_index_callback(ItemPointer itemptr, bool is_warm, void *opaque);
static void validate_index_heapscan(Relation heapRelation,
Relation indexRelation,
IndexInfo *indexInfo,
@@ -1691,6 +1692,20 @@ BuildIndexInfo(Relation index)
ii->ii_AmCache = NULL;
ii->ii_Context = CurrentMemoryContext;
+ /* build a bitmap of all table attributes referred by this index */
+ for (i = 0; i < ii->ii_NumIndexAttrs; i++)
+ {
+ AttrNumber attr = ii->ii_KeyAttrNumbers[i];
+ ii->ii_indxattrs = bms_add_member(ii->ii_indxattrs, attr -
+ FirstLowInvalidHeapAttributeNumber);
+ }
+
+ /* Collect all attributes used in expressions, too */
+ pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_indxattrs);
+
+ /* Collect all attributes in the index predicate, too */
+ pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_indxattrs);
+
return ii;
}
@@ -1816,6 +1831,50 @@ FormIndexDatum(IndexInfo *indexInfo,
elog(ERROR, "wrong number of index expressions");
}
+/*
+ * This is same as FormIndexDatum but we don't compute any expression
+ * attributes and hence can be used when executor interfaces are not available.
+ * If i'th attribute is available then isavail[i] is set to true, else set to
+ * false. The caller must always check if an attribute value is available
+ * before trying to do anything useful with that.
+ */
+void
+FormIndexPlainDatum(IndexInfo *indexInfo,
+ Relation heapRel,
+ HeapTuple heapTup,
+ Datum *values,
+ bool *isnull,
+ bool *isavail)
+{
+ int i;
+
+ for (i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
+ {
+ int keycol = indexInfo->ii_KeyAttrNumbers[i];
+ Datum iDatum;
+ bool isNull;
+
+ if (keycol != 0)
+ {
+ /*
+ * Plain index column; get the value we need directly from the
+ * heap tuple.
+ */
+ iDatum = heap_getattr(heapTup, keycol, RelationGetDescr(heapRel), &isNull);
+ values[i] = iDatum;
+ isnull[i] = isNull;
+ isavail[i] = true;
+ }
+ else
+ {
+ /*
+ * This is an expression attribute which can't be computed by us.
+ * So just inform the caller about it.
+ */
+ isavail[i] = false;
+ }
+ }
+}
/*
* index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX
@@ -2934,15 +2993,15 @@ itemptr_decode(ItemPointer itemptr, int64 encoded)
/*
* validate_index_callback - bulkdelete callback to collect the index TIDs
*/
-static bool
-validate_index_callback(ItemPointer itemptr, void *opaque)
+static IndexBulkDeleteCallbackResult
+validate_index_callback(ItemPointer itemptr, bool is_warm, void *opaque)
{
v_i_state *state = (v_i_state *) opaque;
int64 encoded = itemptr_encode(itemptr);
tuplesort_putdatum(state->tuplesort, Int64GetDatum(encoded), false);
state->itups += 1;
- return false; /* never actually delete anything */
+ return IBDCR_KEEP; /* never actually delete anything */
}
/*
@@ -3163,7 +3222,8 @@ validate_index_heapscan(Relation heapRelation,
heapRelation,
indexInfo->ii_Unique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- indexInfo);
+ indexInfo,
+ false);
state->tups_inserted += 1;
}
diff --git b/src/backend/catalog/indexing.c a/src/backend/catalog/indexing.c
index abc344a..6392f33 100644
--- b/src/backend/catalog/indexing.c
+++ a/src/backend/catalog/indexing.c
@@ -66,10 +66,15 @@ CatalogCloseIndexes(CatalogIndexState indstate)
*
* This should be called for each inserted or updated catalog tuple.
*
+ * If the tuple was WARM updated, the modified_attrs contains the list of
+ * columns updated by the update. We must not insert new index entries for
+ * indexes which do not refer to any of the modified columns.
+ *
* This is effectively a cut-down version of ExecInsertIndexTuples.
*/
static void
-CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
+CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple,
+ Bitmapset *modified_attrs, bool warm_update)
{
int i;
int numIndexes;
@@ -79,12 +84,28 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
IndexInfo **indexInfoArray;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
+ ItemPointerData root_tid;
- /* HOT update does not require index inserts */
- if (HeapTupleIsHeapOnly(heapTuple))
+ /*
+ * HOT update does not require index inserts, but WARM may need for some
+ * indexes.
+ */
+ if (HeapTupleIsHeapOnly(heapTuple) && !warm_update)
return;
/*
+ * If we've done a WARM update, then we must index the TID of the root line
+ * pointer and not the actual TID of the new tuple.
+ */
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(heapTuple->t_self)),
+ HeapTupleHeaderGetRootOffset(heapTuple->t_data));
+ else
+ ItemPointerCopy(&heapTuple->t_self, &root_tid);
+
+
+ /*
* Get information from the state structure. Fall out if nothing to do.
*/
numIndexes = indstate->ri_NumIndices;
@@ -112,6 +133,17 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
continue;
/*
+ * If we've done WARM update, then we must not insert a new index tuple
+ * if none of the index keys have changed. This is not just an
+ * optimization, but a requirement for WARM to work correctly.
+ */
+ if (warm_update)
+ {
+ if (!bms_overlap(modified_attrs, indexInfo->ii_indxattrs))
+ continue;
+ }
+
+ /*
* Expressional and partial indexes on system catalogs are not
* supported, nor exclusion constraints, nor deferred uniqueness
*/
@@ -136,11 +168,12 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple)
index_insert(relationDescs[i], /* index relation */
values, /* array of index Datums */
isnull, /* is-null flags */
- &(heapTuple->t_self), /* tid of heap tuple */
+ &root_tid,
heapRelation,
relationDescs[i]->rd_index->indisunique ?
UNIQUE_CHECK_YES : UNIQUE_CHECK_NO,
- indexInfo);
+ indexInfo,
+ warm_update);
}
ExecDropSingleTupleTableSlot(slot);
@@ -168,7 +201,7 @@ CatalogTupleInsert(Relation heapRel, HeapTuple tup)
oid = simple_heap_insert(heapRel, tup);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, NULL, false);
CatalogCloseIndexes(indstate);
return oid;
@@ -190,7 +223,7 @@ CatalogTupleInsertWithInfo(Relation heapRel, HeapTuple tup,
oid = simple_heap_insert(heapRel, tup);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, NULL, false);
return oid;
}
@@ -210,12 +243,14 @@ void
CatalogTupleUpdate(Relation heapRel, ItemPointer otid, HeapTuple tup)
{
CatalogIndexState indstate;
+ bool warm_update;
+ Bitmapset *modified_attrs;
indstate = CatalogOpenIndexes(heapRel);
- simple_heap_update(heapRel, otid, tup);
+ simple_heap_update(heapRel, otid, tup, &modified_attrs, &warm_update);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, modified_attrs, warm_update);
CatalogCloseIndexes(indstate);
}
@@ -231,9 +266,12 @@ void
CatalogTupleUpdateWithInfo(Relation heapRel, ItemPointer otid, HeapTuple tup,
CatalogIndexState indstate)
{
- simple_heap_update(heapRel, otid, tup);
+ Bitmapset *modified_attrs;
+ bool warm_update;
+
+ simple_heap_update(heapRel, otid, tup, &modified_attrs, &warm_update);
- CatalogIndexInsert(indstate, tup);
+ CatalogIndexInsert(indstate, tup, modified_attrs, warm_update);
}
/*
diff --git b/src/backend/catalog/system_views.sql a/src/backend/catalog/system_views.sql
index b6552da..15d0fe4 100644
--- b/src/backend/catalog/system_views.sql
+++ a/src/backend/catalog/system_views.sql
@@ -498,6 +498,7 @@ CREATE VIEW pg_stat_all_tables AS
pg_stat_get_tuples_updated(C.oid) AS n_tup_upd,
pg_stat_get_tuples_deleted(C.oid) AS n_tup_del,
pg_stat_get_tuples_hot_updated(C.oid) AS n_tup_hot_upd,
+ pg_stat_get_tuples_warm_updated(C.oid) AS n_tup_warm_upd,
pg_stat_get_live_tuples(C.oid) AS n_live_tup,
pg_stat_get_dead_tuples(C.oid) AS n_dead_tup,
pg_stat_get_mod_since_analyze(C.oid) AS n_mod_since_analyze,
@@ -528,7 +529,8 @@ CREATE VIEW pg_stat_xact_all_tables AS
pg_stat_get_xact_tuples_inserted(C.oid) AS n_tup_ins,
pg_stat_get_xact_tuples_updated(C.oid) AS n_tup_upd,
pg_stat_get_xact_tuples_deleted(C.oid) AS n_tup_del,
- pg_stat_get_xact_tuples_hot_updated(C.oid) AS n_tup_hot_upd
+ pg_stat_get_xact_tuples_hot_updated(C.oid) AS n_tup_hot_upd,
+ pg_stat_get_xact_tuples_warm_updated(C.oid) AS n_tup_warm_upd
FROM pg_class C LEFT JOIN
pg_index I ON C.oid = I.indrelid
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git b/src/backend/commands/constraint.c a/src/backend/commands/constraint.c
index e2544e5..330b661 100644
--- b/src/backend/commands/constraint.c
+++ a/src/backend/commands/constraint.c
@@ -40,6 +40,7 @@ unique_key_recheck(PG_FUNCTION_ARGS)
TriggerData *trigdata = castNode(TriggerData, fcinfo->context);
const char *funcname = "unique_key_recheck";
HeapTuple new_row;
+ HeapTupleData heapTuple;
ItemPointerData tmptid;
Relation indexRel;
IndexInfo *indexInfo;
@@ -102,7 +103,8 @@ unique_key_recheck(PG_FUNCTION_ARGS)
* removed.
*/
tmptid = new_row->t_self;
- if (!heap_hot_search(&tmptid, trigdata->tg_relation, SnapshotSelf, NULL))
+ if (!heap_hot_search(&tmptid, trigdata->tg_relation, SnapshotSelf, NULL,
+ NULL, NULL, &heapTuple))
{
/*
* All rows in the HOT chain are dead, so skip the check.
@@ -166,7 +168,8 @@ unique_key_recheck(PG_FUNCTION_ARGS)
*/
index_insert(indexRel, values, isnull, &(new_row->t_self),
trigdata->tg_relation, UNIQUE_CHECK_EXISTING,
- indexInfo);
+ indexInfo,
+ false);
}
else
{
diff --git b/src/backend/commands/copy.c a/src/backend/commands/copy.c
index ba89b29..120e261 100644
--- b/src/backend/commands/copy.c
+++ a/src/backend/commands/copy.c
@@ -2681,6 +2681,8 @@ CopyFrom(CopyState cstate)
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot,
&(tuple->t_self),
+ &(tuple->t_self),
+ NULL,
estate,
false,
NULL,
@@ -2835,6 +2837,7 @@ CopyFromInsertBatch(CopyState cstate, EState *estate, CommandId mycid,
ExecStoreTuple(bufferedTuples[i], myslot, InvalidBuffer, false);
recheckIndexes =
ExecInsertIndexTuples(myslot, &(bufferedTuples[i]->t_self),
+ &(bufferedTuples[i]->t_self), NULL,
estate, false, NULL, NIL);
ExecARInsertTriggers(estate, resultRelInfo,
bufferedTuples[i],
diff --git b/src/backend/commands/indexcmds.c a/src/backend/commands/indexcmds.c
index 9618032..1b2abd4 100644
--- b/src/backend/commands/indexcmds.c
+++ a/src/backend/commands/indexcmds.c
@@ -694,7 +694,14 @@ DefineIndex(Oid relationId,
* visible to other transactions before we start to build the index. That
* will prevent them from making incompatible HOT updates. The new index
* will be marked not indisready and not indisvalid, so that no one else
- * tries to either insert into it or use it for queries.
+ * tries to either insert into it or use it for queries. In addition to
+ * that, WARM updates will be disallowed if an update is modifying one of
+ * the columns used by this new index. This is necessary to ensure that we
+ * don't create WARM tuples which do not have corresponding entry in this
+ * index. It must be noted that during the second phase, we will index only
+ * those heap tuples whose root line pointer is not already in the index,
+ * hence it's important that all tuples in a given chain, has the same
+ * value for any indexed column (including this new index).
*
* We must commit our current transaction so that the index becomes
* visible; then start another. Note that all the data structures we just
@@ -742,7 +749,10 @@ DefineIndex(Oid relationId,
* marked as "not-ready-for-inserts". The index is consulted while
* deciding HOT-safety though. This arrangement ensures that no new HOT
* chains can be created where the new tuple and the old tuple in the
- * chain have different index keys.
+ * chain have different index keys. Also, the new index is consulted for
+ * deciding whether a WARM update is possible, and WARM update is not done
+ * if a column used by this index is being updated. This ensures that we
+ * don't create WARM tuples which are not indexed by this index.
*
* We now take a new snapshot, and build the index using all tuples that
* are visible in this snapshot. We can be sure that any HOT updates to
@@ -777,7 +787,8 @@ DefineIndex(Oid relationId,
/*
* Update the pg_index row to mark the index as ready for inserts. Once we
* commit this transaction, any new transactions that open the table must
- * insert new entries into the index for insertions and non-HOT updates.
+ * insert new entries into the index for insertions and non-HOT updates or
+ * WARM updates where this index needs new entry.
*/
index_set_state_flags(indexRelationId, INDEX_CREATE_SET_READY);
diff --git b/src/backend/commands/vacuumlazy.c a/src/backend/commands/vacuumlazy.c
index b74e493..025a024 100644
--- b/src/backend/commands/vacuumlazy.c
+++ a/src/backend/commands/vacuumlazy.c
@@ -104,6 +104,39 @@
*/
#define PREFETCH_SIZE ((BlockNumber) 32)
+/*
+ * Structure to track WARM chains that can be converted into HOT chains during
+ * this run.
+ *
+ * To reduce space requirement, we're using bitfields. But the way things are
+ * laid down, we're still wasting 1-byte per candidate chain.
+ */
+typedef struct LVWarmChain
+{
+ ItemPointerData chain_tid; /* root of the chain */
+
+ /*
+ * 1 - if the chain contains only post-warm tuples
+ * 0 - if the chain contains only pre-warm tuples
+ */
+ uint8 is_postwarm_chain:2;
+
+ /* 1 - if this chain can't be cleared of WARM tuples */
+ uint8 keep_warm_chain:2;
+
+ /*
+ * Number of CLEAR pointers to this root TID found so far - must never be
+ * more than 2.
+ */
+ uint8 num_clear_pointers:2;
+
+ /*
+ * Number of WARM pointers to this root TID found so far - must never be
+ * more than 1.
+ */
+ uint8 num_warm_pointers:2;
+} LVWarmChain;
+
typedef struct LVRelStats
{
/* hasindex = true means two-pass strategy; false means one-pass */
@@ -122,6 +155,14 @@ typedef struct LVRelStats
BlockNumber pages_removed;
double tuples_deleted;
BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
+
+ /* List of candidate WARM chains that can be converted into HOT chains */
+ /* NB: this list is ordered by TID of the root pointers */
+ int num_warm_chains; /* current # of entries */
+ int max_warm_chains; /* # slots allocated in array */
+ LVWarmChain *warm_chains; /* array of LVWarmChain */
+ double num_non_convertible_warm_chains;
+
/* List of TIDs of tuples we intend to delete */
/* NB: this list is ordered by TID address */
int num_dead_tuples; /* current # of entries */
@@ -150,6 +191,7 @@ static void lazy_scan_heap(Relation onerel, int options,
static void lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats);
static bool lazy_check_needs_freeze(Buffer buf, bool *hastup);
static void lazy_vacuum_index(Relation indrel,
+ bool clear_warm,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats);
static void lazy_cleanup_index(Relation indrel,
@@ -157,6 +199,10 @@ static void lazy_cleanup_index(Relation indrel,
LVRelStats *vacrelstats);
static int lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer,
int tupindex, LVRelStats *vacrelstats, Buffer *vmbuffer);
+static int lazy_warmclear_page(Relation onerel, BlockNumber blkno,
+ Buffer buffer, int chainindex, LVRelStats *vacrelstats,
+ Buffer *vmbuffer, bool check_all_visible);
+static void lazy_reset_warm_pointer_count(LVRelStats *vacrelstats);
static bool should_attempt_truncation(LVRelStats *vacrelstats);
static void lazy_truncate_heap(Relation onerel, LVRelStats *vacrelstats);
static BlockNumber count_nondeletable_pages(Relation onerel,
@@ -164,8 +210,15 @@ static BlockNumber count_nondeletable_pages(Relation onerel,
static void lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks);
static void lazy_record_dead_tuple(LVRelStats *vacrelstats,
ItemPointer itemptr);
-static bool lazy_tid_reaped(ItemPointer itemptr, void *state);
+static void lazy_record_warm_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr);
+static void lazy_record_clear_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr);
+static IndexBulkDeleteCallbackResult lazy_tid_reaped(ItemPointer itemptr, bool is_warm, void *state);
+static IndexBulkDeleteCallbackResult lazy_indexvac_phase1(ItemPointer itemptr, bool is_warm, void *state);
+static IndexBulkDeleteCallbackResult lazy_indexvac_phase2(ItemPointer itemptr, bool is_warm, void *state);
static int vac_cmp_itemptr(const void *left, const void *right);
+static int vac_cmp_warm_chain(const void *left, const void *right);
static bool heap_page_is_all_visible(Relation rel, Buffer buf,
TransactionId *visibility_cutoff_xid, bool *all_frozen);
@@ -690,8 +743,10 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* If we are close to overrunning the available space for dead-tuple
* TIDs, pause and do a cycle of vacuuming before we tackle this page.
*/
- if ((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
- vacrelstats->num_dead_tuples > 0)
+ if (((vacrelstats->max_dead_tuples - vacrelstats->num_dead_tuples) < MaxHeapTuplesPerPage &&
+ vacrelstats->num_dead_tuples > 0) ||
+ ((vacrelstats->max_warm_chains - vacrelstats->num_warm_chains) < MaxHeapTuplesPerPage &&
+ vacrelstats->num_warm_chains > 0))
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
@@ -721,6 +776,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
+ (vacrelstats->num_warm_chains > 0),
&indstats[i],
vacrelstats);
@@ -743,6 +799,9 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
* valid.
*/
vacrelstats->num_dead_tuples = 0;
+ vacrelstats->num_warm_chains = 0;
+ memset(vacrelstats->warm_chains, 0,
+ vacrelstats->max_warm_chains * sizeof (LVWarmChain));
vacrelstats->num_index_scans++;
/* Report that we are once again scanning the heap */
@@ -947,15 +1006,31 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
continue;
}
+ ItemPointerSet(&(tuple.t_self), blkno, offnum);
+
/* Redirect items mustn't be touched */
if (ItemIdIsRedirected(itemid))
{
+ HeapCheckWarmChainStatus status = heap_check_warm_chain(page,
+ &tuple.t_self, false);
+ if (HCWC_IS_WARM_UPDATED(status))
+ {
+ /*
+ * A chain which is either complete WARM or CLEAR is a
+ * candidate for chain conversion. Remember the chain and
+ * whether the chain has all WARM tuples or not.
+ */
+ if (HCWC_IS_ALL_WARM(status))
+ lazy_record_warm_chain(vacrelstats, &tuple.t_self);
+ else if (HCWC_IS_ALL_CLEAR(status))
+ lazy_record_clear_chain(vacrelstats, &tuple.t_self);
+ else
+ vacrelstats->num_non_convertible_warm_chains++;
+ }
hastup = true; /* this page won't be truncatable */
continue;
}
- ItemPointerSet(&(tuple.t_self), blkno, offnum);
-
/*
* DEAD item pointers are to be vacuumed normally; but we don't
* count them in tups_vacuumed, else we'd be double-counting (at
@@ -975,6 +1050,26 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
tuple.t_len = ItemIdGetLength(itemid);
tuple.t_tableOid = RelationGetRelid(onerel);
+ if (!HeapTupleIsHeapOnly(&tuple))
+ {
+ HeapCheckWarmChainStatus status = heap_check_warm_chain(page,
+ &tuple.t_self, false);
+ if (HCWC_IS_WARM_UPDATED(status))
+ {
+ /*
+ * A chain which is either complete WARM or CLEAR is a
+ * candidate for chain conversion. Remember the chain and
+ * its color.
+ */
+ if (HCWC_IS_ALL_WARM(status))
+ lazy_record_warm_chain(vacrelstats, &tuple.t_self);
+ else if (HCWC_IS_ALL_CLEAR(status))
+ lazy_record_clear_chain(vacrelstats, &tuple.t_self);
+ else
+ vacrelstats->num_non_convertible_warm_chains++;
+ }
+ }
+
tupgone = false;
switch (HeapTupleSatisfiesVacuum(&tuple, OldestXmin, buf))
@@ -1040,6 +1135,19 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
break;
}
+ /*
+ * If this tuple was ever WARM updated or is a WARM
+ * tuple, there could be multiple index entries
+ * pointing to the root of this chain. We can't do
+ * index-only scans for such tuples without verifying
+ * index key check. So mark the page as !all_visible
+ */
+ if (HeapTupleHeaderIsWarmUpdated(tuple.t_data))
+ {
+ all_visible = false;
+ break;
+ }
+
/* Track newest xmin on page. */
if (TransactionIdFollows(xmin, visibility_cutoff_xid))
visibility_cutoff_xid = xmin;
@@ -1282,7 +1390,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* If any tuples need to be deleted, perform final vacuum cycle */
/* XXX put a threshold on min number of tuples here? */
- if (vacrelstats->num_dead_tuples > 0)
+ if (vacrelstats->num_dead_tuples > 0 || vacrelstats->num_warm_chains > 0)
{
const int hvp_index[] = {
PROGRESS_VACUUM_PHASE,
@@ -1300,6 +1408,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
/* Remove index entries */
for (i = 0; i < nindexes; i++)
lazy_vacuum_index(Irel[i],
+ (vacrelstats->num_warm_chains > 0),
&indstats[i],
vacrelstats);
@@ -1367,7 +1476,10 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
*
* This routine marks dead tuples as unused and compacts out free
* space on their pages. Pages not having dead tuples recorded from
- * lazy_scan_heap are not visited at all.
+ * lazy_scan_heap are not visited at all. This routine also converts
+ * candidate WARM chains to HOT chains by clearing WARM related flags. The
+ * candidate chains are determined by the preceeding index scans after
+ * looking at the data collected by the first heap scan.
*
* Note: the reason for doing this as a second pass is we cannot remove
* the tuples until we've removed their index entries, and we want to
@@ -1376,7 +1488,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
static void
lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
{
- int tupindex;
+ int tupindex, chainindex;
int npages;
PGRUsage ru0;
Buffer vmbuffer = InvalidBuffer;
@@ -1385,33 +1497,69 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
npages = 0;
tupindex = 0;
- while (tupindex < vacrelstats->num_dead_tuples)
+ chainindex = 0;
+ while (tupindex < vacrelstats->num_dead_tuples ||
+ chainindex < vacrelstats->num_warm_chains)
{
- BlockNumber tblk;
+ BlockNumber tblk, chainblk, vacblk;
Buffer buf;
Page page;
Size freespace;
vacuum_delay_point();
- tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
- buf = ReadBufferExtended(onerel, MAIN_FORKNUM, tblk, RBM_NORMAL,
+ tblk = chainblk = InvalidBlockNumber;
+ if (chainindex < vacrelstats->num_warm_chains)
+ chainblk =
+ ItemPointerGetBlockNumber(&(vacrelstats->warm_chains[chainindex].chain_tid));
+
+ if (tupindex < vacrelstats->num_dead_tuples)
+ tblk = ItemPointerGetBlockNumber(&vacrelstats->dead_tuples[tupindex]);
+
+ if (tblk == InvalidBlockNumber)
+ vacblk = chainblk;
+ else if (chainblk == InvalidBlockNumber)
+ vacblk = tblk;
+ else
+ vacblk = Min(chainblk, tblk);
+
+ Assert(vacblk != InvalidBlockNumber);
+
+ buf = ReadBufferExtended(onerel, MAIN_FORKNUM, vacblk, RBM_NORMAL,
vac_strategy);
- if (!ConditionalLockBufferForCleanup(buf))
+
+
+ if (vacblk == chainblk)
+ LockBufferForCleanup(buf);
+ else if (!ConditionalLockBufferForCleanup(buf))
{
ReleaseBuffer(buf);
++tupindex;
continue;
}
- tupindex = lazy_vacuum_page(onerel, tblk, buf, tupindex, vacrelstats,
- &vmbuffer);
+
+ /*
+ * Convert WARM chains on this page. This should be done before
+ * vacuuming the page to ensure that we can correctly set visibility
+ * bits after clearing WARM chains.
+ *
+ * If we are going to vacuum this page then don't check for
+ * all-visibility just yet.
+ */
+ if (vacblk == chainblk)
+ chainindex = lazy_warmclear_page(onerel, chainblk, buf, chainindex,
+ vacrelstats, &vmbuffer, chainblk != tblk);
+
+ if (vacblk == tblk)
+ tupindex = lazy_vacuum_page(onerel, tblk, buf, tupindex, vacrelstats,
+ &vmbuffer);
/* Now that we've compacted the page, record its available space */
page = BufferGetPage(buf);
freespace = PageGetHeapFreeSpace(page);
UnlockReleaseBuffer(buf);
- RecordPageWithFreeSpace(onerel, tblk, freespace);
+ RecordPageWithFreeSpace(onerel, vacblk, freespace);
npages++;
}
@@ -1430,6 +1578,107 @@ lazy_vacuum_heap(Relation onerel, LVRelStats *vacrelstats)
}
/*
+ * lazy_warmclear_page() -- clear various WARM bits on the tuples.
+ *
+ * Caller must hold pin and buffer cleanup lock on the buffer.
+ *
+ * chainindex is the index in vacrelstats->warm_chains of the first dead
+ * tuple for this page. We assume the rest follow sequentially.
+ * The return value is the first tupindex after the tuples of this page.
+ *
+ * If check_all_visible is set then we also check if the page has now become
+ * all visible and update visibility map.
+ */
+static int
+lazy_warmclear_page(Relation onerel, BlockNumber blkno, Buffer buffer,
+ int chainindex, LVRelStats *vacrelstats, Buffer *vmbuffer,
+ bool check_all_visible)
+{
+ Page page = BufferGetPage(buffer);
+ OffsetNumber cleared_offnums[MaxHeapTuplesPerPage];
+ int num_cleared = 0;
+ TransactionId visibility_cutoff_xid;
+ bool all_frozen;
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_WARMCLEARED, blkno);
+
+ START_CRIT_SECTION();
+
+ for (; chainindex < vacrelstats->num_warm_chains ; chainindex++)
+ {
+ BlockNumber tblk;
+ LVWarmChain *chain;
+
+ chain = &vacrelstats->warm_chains[chainindex];
+
+ tblk = ItemPointerGetBlockNumber(&chain->chain_tid);
+ if (tblk != blkno)
+ break; /* past end of tuples for this block */
+
+ /*
+ * Since a heap page can have no more than MaxHeapTuplesPerPage
+ * offnums and we process each offnum only once, MaxHeapTuplesPerPage
+ * size array should be enough to hold all cleared tuples in this page.
+ */
+ if (!chain->keep_warm_chain)
+ num_cleared += heap_clear_warm_chain(page, &chain->chain_tid,
+ cleared_offnums + num_cleared);
+ }
+
+ /*
+ * Mark buffer dirty before we write WAL.
+ */
+ MarkBufferDirty(buffer);
+
+ /* XLOG stuff */
+ if (RelationNeedsWAL(onerel))
+ {
+ XLogRecPtr recptr;
+
+ recptr = log_heap_warmclear(onerel, buffer,
+ cleared_offnums, num_cleared);
+ PageSetLSN(page, recptr);
+ }
+
+ END_CRIT_SECTION();
+
+ /* If not checking for all-visibility then we're done */
+ if (!check_all_visible)
+ return chainindex;
+
+ /*
+ * The following code should match the corresponding code in
+ * lazy_vacuum_page
+ **/
+ if (heap_page_is_all_visible(onerel, buffer, &visibility_cutoff_xid,
+ &all_frozen))
+ PageSetAllVisible(page);
+
+ /*
+ * All the changes to the heap page have been done. If the all-visible
+ * flag is now set, also set the VM all-visible bit (and, if possible, the
+ * all-frozen bit) unless this has already been done previously.
+ */
+ if (PageIsAllVisible(page))
+ {
+ uint8 vm_status = visibilitymap_get_status(onerel, blkno, vmbuffer);
+ uint8 flags = 0;
+
+ /* Set the VM all-frozen bit to flag, if needed */
+ if ((vm_status & VISIBILITYMAP_ALL_VISIBLE) == 0)
+ flags |= VISIBILITYMAP_ALL_VISIBLE;
+ if ((vm_status & VISIBILITYMAP_ALL_FROZEN) == 0 && all_frozen)
+ flags |= VISIBILITYMAP_ALL_FROZEN;
+
+ Assert(BufferIsValid(*vmbuffer));
+ if (flags != 0)
+ visibilitymap_set(onerel, blkno, buffer, InvalidXLogRecPtr,
+ *vmbuffer, visibility_cutoff_xid, flags);
+ }
+ return chainindex;
+}
+
+/*
* lazy_vacuum_page() -- free dead tuples on a page
* and repair its fragmentation.
*
@@ -1582,6 +1831,24 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
return false;
}
+/*
+ * Reset counters tracking number of WARM and CLEAR pointers per candidate TID.
+ * These counters are maintained per index and cleared when the next index is
+ * picked up for cleanup.
+ *
+ * We don't touch the keep_warm_chain since once a chain is known to be
+ * non-convertible, we must remember that across all indexes.
+ */
+static void
+lazy_reset_warm_pointer_count(LVRelStats *vacrelstats)
+{
+ int i;
+ for (i = 0; i < vacrelstats->num_warm_chains; i++)
+ {
+ LVWarmChain *chain = &vacrelstats->warm_chains[i];
+ chain->num_clear_pointers = chain->num_warm_pointers = 0;
+ }
+}
/*
* lazy_vacuum_index() -- vacuum one index relation.
@@ -1591,6 +1858,7 @@ lazy_check_needs_freeze(Buffer buf, bool *hastup)
*/
static void
lazy_vacuum_index(Relation indrel,
+ bool clear_warm,
IndexBulkDeleteResult **stats,
LVRelStats *vacrelstats)
{
@@ -1606,15 +1874,87 @@ lazy_vacuum_index(Relation indrel,
ivinfo.num_heap_tuples = vacrelstats->old_rel_tuples;
ivinfo.strategy = vac_strategy;
- /* Do bulk deletion */
- *stats = index_bulk_delete(&ivinfo, *stats,
- lazy_tid_reaped, (void *) vacrelstats);
+ /*
+ * If told, convert WARM chains into HOT chains.
+ *
+ * We must have already collected candidate WARM chains i.e. chains that
+ * have either all tuples with HEAP_WARM_TUPLE flag set or none.
+ *
+ * This works in two phases. In the first phase, we do a complete index
+ * scan and collect information about index pointers to the candidate
+ * chains, but we don't do conversion. To be precise, we count the number
+ * of WARM and CLEAR index pointers to each candidate chain and use that
+ * knowledge to arrive at a decision and do the actual conversion during
+ * the second phase (we kill known dead pointers though in this phase).
+ *
+ * In the second phase, for each candidate chain we check if we have seen a
+ * WARM index pointer. For such chains, we kill the CLEAR pointer and
+ * convert the WARM pointer into a CLEAR pointer. The heap tuples are
+ * cleared of WARM flags in the second heap scan. If we did not find any
+ * WARM pointer to a WARM chain, that means that the chain is reachable
+ * from the CLEAR pointer (because say WARM update did not add a new entry
+ * for this index). In that case, we do nothing. There is a third case
+ * where we find two CLEAR pointers to a candidate chain. This can happen
+ * because of aborted vacuums. We don't handle that case yet, but it should
+ * be possible to apply the same recheck logic and find which of the clear
+ * pointers is redundant and should be removed.
+ *
+ * For CLEAR chains, we just kill the WARM pointer, if it exists, and keep
+ * the CLEAR pointer.
+ */
+ if (clear_warm)
+ {
+ /*
+ * Before starting the index scan, reset the counters of WARM and CLEAR
+ * pointers, probably carried forward from the previous index.
+ */
+ lazy_reset_warm_pointer_count(vacrelstats);
+
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_indexvac_phase1, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to remove %d row version, found "
+ "%0.f warm pointers, %0.f clear pointers, removed "
+ "%0.f warm pointers, removed %0.f clear pointers",
+ RelationGetRelationName(indrel),
+ vacrelstats->num_dead_tuples,
+ (*stats)->num_warm_pointers,
+ (*stats)->num_clear_pointers,
+ (*stats)->warm_pointers_removed,
+ (*stats)->clear_pointers_removed)));
+
+ (*stats)->num_warm_pointers = 0;
+ (*stats)->num_clear_pointers = 0;
+ (*stats)->warm_pointers_removed = 0;
+ (*stats)->clear_pointers_removed = 0;
+ (*stats)->pointers_cleared = 0;
+
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_indexvac_phase2, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to convert WARM pointers, found "
+ "%0.f WARM pointers, %0.f CLEAR pointers, removed "
+ "%0.f WARM pointers, removed %0.f CLEAR pointers, "
+ "cleared %0.f WARM pointers",
+ RelationGetRelationName(indrel),
+ (*stats)->num_warm_pointers,
+ (*stats)->num_clear_pointers,
+ (*stats)->warm_pointers_removed,
+ (*stats)->clear_pointers_removed,
+ (*stats)->pointers_cleared)));
+ }
+ else
+ {
+ /* Do bulk deletion */
+ *stats = index_bulk_delete(&ivinfo, *stats,
+ lazy_tid_reaped, (void *) vacrelstats);
+ ereport(elevel,
+ (errmsg("scanned index \"%s\" to remove %d row versions",
+ RelationGetRelationName(indrel),
+ vacrelstats->num_dead_tuples),
+ errdetail("%s.", pg_rusage_show(&ru0))));
+ }
- ereport(elevel,
- (errmsg("scanned index \"%s\" to remove %d row versions",
- RelationGetRelationName(indrel),
- vacrelstats->num_dead_tuples),
- errdetail("%s.", pg_rusage_show(&ru0))));
}
/*
@@ -1988,9 +2328,11 @@ lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks)
if (vacrelstats->hasindex)
{
- maxtuples = (vac_work_mem * 1024L) / sizeof(ItemPointerData);
+ maxtuples = (vac_work_mem * 1024L) / (sizeof(ItemPointerData) +
+ sizeof(LVWarmChain));
maxtuples = Min(maxtuples, INT_MAX);
- maxtuples = Min(maxtuples, MaxAllocSize / sizeof(ItemPointerData));
+ maxtuples = Min(maxtuples, MaxAllocSize / (sizeof(ItemPointerData) +
+ sizeof(LVWarmChain)));
/* curious coding here to ensure the multiplication can't overflow */
if ((BlockNumber) (maxtuples / LAZY_ALLOC_TUPLES) > relblocks)
@@ -2008,6 +2350,57 @@ lazy_space_alloc(LVRelStats *vacrelstats, BlockNumber relblocks)
vacrelstats->max_dead_tuples = (int) maxtuples;
vacrelstats->dead_tuples = (ItemPointer)
palloc(maxtuples * sizeof(ItemPointerData));
+
+ /*
+ * XXX Cheat for now and allocate the same size array for tracking warm
+ * chains. maxtuples must have been already adjusted above to ensure we
+ * don't cross vac_work_mem.
+ */
+ vacrelstats->num_warm_chains = 0;
+ vacrelstats->max_warm_chains = (int) maxtuples;
+ vacrelstats->warm_chains = (LVWarmChain *)
+ palloc0(maxtuples * sizeof(LVWarmChain));
+
+}
+
+/*
+ * lazy_record_clear_chain - remember one CLEAR chain
+ */
+static void
+lazy_record_clear_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr)
+{
+ /*
+ * The array shouldn't overflow under normal behavior, but perhaps it
+ * could if we are given a really small maintenance_work_mem. In that
+ * case, just forget the last few tuples (we'll get 'em next time).
+ */
+ if (vacrelstats->num_warm_chains < vacrelstats->max_warm_chains)
+ {
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].chain_tid = *itemptr;
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].is_postwarm_chain = 0;
+ vacrelstats->num_warm_chains++;
+ }
+}
+
+/*
+ * lazy_record_warm_chain - remember one WARM chain
+ */
+static void
+lazy_record_warm_chain(LVRelStats *vacrelstats,
+ ItemPointer itemptr)
+{
+ /*
+ * The array shouldn't overflow under normal behavior, but perhaps it
+ * could if we are given a really small maintenance_work_mem. In that
+ * case, just forget the last few tuples (we'll get 'em next time).
+ */
+ if (vacrelstats->num_warm_chains < vacrelstats->max_warm_chains)
+ {
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].chain_tid = *itemptr;
+ vacrelstats->warm_chains[vacrelstats->num_warm_chains].is_postwarm_chain = 1;
+ vacrelstats->num_warm_chains++;
+ }
}
/*
@@ -2038,8 +2431,8 @@ lazy_record_dead_tuple(LVRelStats *vacrelstats,
*
* Assumes dead_tuples array is in sorted order.
*/
-static bool
-lazy_tid_reaped(ItemPointer itemptr, void *state)
+static IndexBulkDeleteCallbackResult
+lazy_tid_reaped(ItemPointer itemptr, bool is_warm, void *state)
{
LVRelStats *vacrelstats = (LVRelStats *) state;
ItemPointer res;
@@ -2050,7 +2443,193 @@ lazy_tid_reaped(ItemPointer itemptr, void *state)
sizeof(ItemPointerData),
vac_cmp_itemptr);
- return (res != NULL);
+ return (res != NULL) ? IBDCR_DELETE : IBDCR_KEEP;
+}
+
+/*
+ * lazy_indexvac_phase1() -- run first pass of index vacuum
+ *
+ * This has the right signature to be an IndexBulkDeleteCallback.
+ */
+static IndexBulkDeleteCallbackResult
+lazy_indexvac_phase1(ItemPointer itemptr, bool is_warm, void *state)
+{
+ LVRelStats *vacrelstats = (LVRelStats *) state;
+ ItemPointer res;
+ LVWarmChain *chain;
+
+ res = (ItemPointer) bsearch((void *) itemptr,
+ (void *) vacrelstats->dead_tuples,
+ vacrelstats->num_dead_tuples,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+
+ if (res != NULL)
+ return IBDCR_DELETE;
+
+ chain = (LVWarmChain *) bsearch((void *) itemptr,
+ (void *) vacrelstats->warm_chains,
+ vacrelstats->num_warm_chains,
+ sizeof(LVWarmChain),
+ vac_cmp_warm_chain);
+ if (chain != NULL)
+ {
+ if (is_warm)
+ chain->num_warm_pointers++;
+ else
+ chain->num_clear_pointers++;
+ }
+ return IBDCR_KEEP;
+}
+
+/*
+ * lazy_indexvac_phase2() -- run second pass of index vacuum
+ *
+ * This has the right signature to be an IndexBulkDeleteCallback.
+ */
+static IndexBulkDeleteCallbackResult
+lazy_indexvac_phase2(ItemPointer itemptr, bool is_warm, void *state)
+{
+ LVRelStats *vacrelstats = (LVRelStats *) state;
+ LVWarmChain *chain;
+
+ chain = (LVWarmChain *) bsearch((void *) itemptr,
+ (void *) vacrelstats->warm_chains,
+ vacrelstats->num_warm_chains,
+ sizeof(LVWarmChain),
+ vac_cmp_warm_chain);
+
+ if (chain != NULL && (chain->keep_warm_chain != 1))
+ {
+ /*
+ * At no point, we can have more than 1 warm pointer to any chain and
+ * no more than 2 clear pointers.
+ */
+ Assert(chain->num_warm_pointers <= 1);
+ Assert(chain->num_clear_pointers <= 2);
+
+ if (chain->is_postwarm_chain == 1)
+ {
+ if (is_warm)
+ {
+ /*
+ * A WARM pointer, pointing to a WARM chain.
+ *
+ * Clear the warm pointer (and delete the CLEAR pointer). We
+ * may have already seen the CLEAR pointer in the scan and
+ * deleted that or we may see it later in the scan. It doesn't
+ * matter if we fail at any point because we won't clear up
+ * WARM bits on the heap tuples until we have dealt with the
+ * index pointers cleanly.
+ */
+ return IBDCR_CLEAR_WARM;
+ }
+ else
+ {
+ /*
+ * CLEAR pointer to a WARM chain.
+ */
+ if (chain->num_warm_pointers > 0)
+ {
+ /*
+ * If there exists a WARM pointer to the chain, we can
+ * delete the CLEAR pointer and clear the WARM bits on the
+ * heap tuples.
+ */
+ return IBDCR_DELETE;
+ }
+ else if (chain->num_clear_pointers == 1)
+ {
+ /*
+ * If this is the only pointer to a WARM chain, we must
+ * keep the CLEAR pointer.
+ *
+ * The presence of WARM chain indicates that the WARM update
+ * must have been committed good. But during the update
+ * this index was probably not updated and hence it
+ * contains just one, original CLEAR pointer to the chain.
+ * We should be able to clear the WARM bits on heap tuples
+ * unless we later find another index which prevents the
+ * cleanup.
+ */
+ return IBDCR_KEEP;
+ }
+ }
+ }
+ else
+ {
+ /*
+ * This is a CLEAR chain.
+ */
+ if (is_warm)
+ {
+ /*
+ * A WARM pointer to a CLEAR chain.
+ *
+ * This can happen when a WARM update is aborted. Later the HOT
+ * chain is pruned leaving behind only CLEAR tuples in the
+ * chain. But the WARM index pointer inserted in the index
+ * remains and it must now be deleted before we clear WARM bits
+ * from the heap tuple.
+ */
+ return IBDCR_DELETE;
+ }
+
+ /*
+ * CLEAR pointer to a CLEAR chain.
+ *
+ * If this is the only surviving CLEAR pointer, keep it and clear
+ * the WARM bits from the heap tuples.
+ */
+ if (chain->num_clear_pointers == 1)
+ return IBDCR_KEEP;
+
+ /*
+ * If there are more than 1 CLEAR pointers to this chain, we can
+ * apply the recheck logic and kill the redudant CLEAR pointer and
+ * convert the chain. But that's not yet done.
+ */
+ }
+
+ /*
+ * For everything else, we must keep the WARM bits and also keep the
+ * index pointers.
+ */
+ chain->keep_warm_chain = 1;
+ return IBDCR_KEEP;
+ }
+ return IBDCR_KEEP;
+}
+
+/*
+ * Comparator routines for use with qsort() and bsearch(). Similar to
+ * vac_cmp_itemptr, but right hand argument is LVWarmChain struct pointer.
+ */
+static int
+vac_cmp_warm_chain(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber(&((LVWarmChain *) right)->chain_tid);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber(&((LVWarmChain *) right)->chain_tid);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
}
/*
@@ -2166,6 +2745,18 @@ heap_page_is_all_visible(Relation rel, Buffer buf,
break;
}
+ /*
+ * If this or any other tuple in the chain ever WARM
+ * updated, there could be multiple index entries pointing
+ * to the root of this chain. We can't do index-only scans
+ * for such tuples without verifying index key check. So
+ * mark the page as !all_visible
+ */
+ if (HeapTupleHeaderIsWarmUpdated(tuple.t_data))
+ {
+ all_visible = false;
+ }
+
/* Track newest xmin on page. */
if (TransactionIdFollows(xmin, *visibility_cutoff_xid))
*visibility_cutoff_xid = xmin;
diff --git b/src/backend/executor/execIndexing.c a/src/backend/executor/execIndexing.c
index 2142273..0beca6d 100644
--- b/src/backend/executor/execIndexing.c
+++ a/src/backend/executor/execIndexing.c
@@ -270,6 +270,8 @@ ExecCloseIndices(ResultRelInfo *resultRelInfo)
List *
ExecInsertIndexTuples(TupleTableSlot *slot,
ItemPointer tupleid,
+ ItemPointer root_tid,
+ Bitmapset *modified_attrs,
EState *estate,
bool noDupErr,
bool *specConflict,
@@ -324,6 +326,17 @@ ExecInsertIndexTuples(TupleTableSlot *slot,
if (!indexInfo->ii_ReadyForInserts)
continue;
+ /*
+ * If modified_attrs is set, we only insert index entries for those
+ * indexes whose column has changed. All other indexes can use their
+ * existing index pointers to look up the new tuple
+ */
+ if (modified_attrs)
+ {
+ if (!bms_overlap(modified_attrs, indexInfo->ii_indxattrs))
+ continue;
+ }
+
/* Check for partial index */
if (indexInfo->ii_Predicate != NIL)
{
@@ -389,10 +402,11 @@ ExecInsertIndexTuples(TupleTableSlot *slot,
index_insert(indexRelation, /* index relation */
values, /* array of index Datums */
isnull, /* null flags */
- tupleid, /* tid of heap tuple */
+ root_tid, /* tid of heap or root tuple */
heapRelation, /* heap relation */
checkUnique, /* type of uniqueness check to do */
- indexInfo); /* index AM may need this */
+ indexInfo, /* index AM may need this */
+ (modified_attrs != NULL)); /* is it a WARM update? */
/*
* If the index has an associated exclusion constraint, check that.
@@ -791,6 +805,9 @@ retry:
{
if (!HeapTupleHeaderIsHeapLatest(tup->t_data, &tup->t_self))
HeapTupleHeaderGetNextTid(tup->t_data, &ctid_wait);
+ else
+ ItemPointerCopy(&tup->t_self, &ctid_wait);
+
reason_wait = indexInfo->ii_ExclusionOps ?
XLTW_RecheckExclusionConstr : XLTW_InsertIndex;
index_endscan(index_scan);
diff --git b/src/backend/executor/execReplication.c a/src/backend/executor/execReplication.c
index f20d728..747e4ce 100644
--- b/src/backend/executor/execReplication.c
+++ a/src/backend/executor/execReplication.c
@@ -399,6 +399,8 @@ ExecSimpleRelationInsert(EState *estate, TupleTableSlot *slot)
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self),
+ NULL,
estate, false, NULL,
NIL);
@@ -445,6 +447,8 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ bool warm_update;
+ Bitmapset *modified_attrs;
/* Check the constraints of the tuple */
if (rel->rd_att->constr)
@@ -455,13 +459,35 @@ ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
/* OK, update the tuple and index entries for it */
simple_heap_update(rel, &searchslot->tts_tuple->t_self,
- slot->tts_tuple);
+ slot->tts_tuple, &modified_attrs, &warm_update);
if (resultRelInfo->ri_NumIndices > 0 &&
- !HeapTupleIsHeapOnly(slot->tts_tuple))
+ (!HeapTupleIsHeapOnly(slot->tts_tuple) || warm_update))
+ {
+ ItemPointerData root_tid;
+
+ /*
+ * If we did a WARM update then we must index the tuple using its
+ * root line pointer and not the tuple TID itself.
+ */
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(tuple->t_self)),
+ HeapTupleHeaderGetRootOffset(tuple->t_data));
+ else
+ {
+ ItemPointerCopy(&tuple->t_self,
+ &root_tid);
+ bms_free(modified_attrs);
+ modified_attrs = NULL;
+ }
+
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &root_tid,
+ modified_attrs,
estate, false, NULL,
NIL);
+ }
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
diff --git b/src/backend/executor/nodeBitmapHeapscan.c a/src/backend/executor/nodeBitmapHeapscan.c
index 2e9ff7d..f7bb6ca 100644
--- b/src/backend/executor/nodeBitmapHeapscan.c
+++ a/src/backend/executor/nodeBitmapHeapscan.c
@@ -39,6 +39,7 @@
#include "access/relscan.h"
#include "access/transam.h"
+#include "access/valid.h"
#include "executor/execdebug.h"
#include "executor/nodeBitmapHeapscan.h"
#include "pgstat.h"
@@ -395,11 +396,27 @@ bitgetpage(HeapScanDesc scan, TBMIterateResult *tbmres)
OffsetNumber offnum = tbmres->offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
+ bool recheck = false;
ItemPointerSet(&tid, page, offnum);
if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot,
- &heapTuple, NULL, true))
- scan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
+ &heapTuple, NULL, true, &recheck))
+ {
+ bool valid = true;
+
+ if (scan->rs_key)
+ HeapKeyTest(&heapTuple, RelationGetDescr(scan->rs_rd),
+ scan->rs_nkeys, scan->rs_key, valid);
+ if (valid)
+ scan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid);
+
+ /*
+ * If the heap tuple needs a recheck because of a WARM update,
+ * it's a lossy case.
+ */
+ if (recheck)
+ tbmres->recheck = true;
+ }
}
}
else
diff --git b/src/backend/executor/nodeIndexscan.c a/src/backend/executor/nodeIndexscan.c
index cb6aff9..dff4086 100644
--- b/src/backend/executor/nodeIndexscan.c
+++ a/src/backend/executor/nodeIndexscan.c
@@ -142,8 +142,8 @@ IndexNext(IndexScanState *node)
false); /* don't pfree */
/*
- * If the index was lossy, we have to recheck the index quals using
- * the fetched tuple.
+ * If the index was lossy or the tuple was WARM, we have to recheck
+ * the index quals using the fetched tuple.
*/
if (scandesc->xs_recheck)
{
diff --git b/src/backend/executor/nodeModifyTable.c a/src/backend/executor/nodeModifyTable.c
index 95e1589..a1f3440 100644
--- b/src/backend/executor/nodeModifyTable.c
+++ a/src/backend/executor/nodeModifyTable.c
@@ -512,6 +512,7 @@ ExecInsert(ModifyTableState *mtstate,
/* insert index entries for tuple */
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self), NULL,
estate, true, &specConflict,
arbiterIndexes);
@@ -558,6 +559,7 @@ ExecInsert(ModifyTableState *mtstate,
/* insert index entries for tuple */
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &(tuple->t_self), NULL,
estate, false, NULL,
arbiterIndexes);
}
@@ -891,6 +893,9 @@ ExecUpdate(ItemPointer tupleid,
HTSU_Result result;
HeapUpdateFailureData hufd;
List *recheckIndexes = NIL;
+ Bitmapset *modified_attrs = NULL;
+ ItemPointerData root_tid;
+ bool warm_update;
/*
* abort the operation if not running transactions
@@ -1007,7 +1012,7 @@ lreplace:;
estate->es_output_cid,
estate->es_crosscheck_snapshot,
true /* wait for commit */ ,
- &hufd, &lockmode);
+ &hufd, &lockmode, &modified_attrs, &warm_update);
switch (result)
{
case HeapTupleSelfUpdated:
@@ -1094,10 +1099,28 @@ lreplace:;
* the t_self field.
*
* If it's a HOT update, we mustn't insert new index entries.
+ *
+ * If it's a WARM update, then we must insert new entries with TID
+ * pointing to the root of the WARM chain.
*/
- if (resultRelInfo->ri_NumIndices > 0 && !HeapTupleIsHeapOnly(tuple))
+ if (resultRelInfo->ri_NumIndices > 0 &&
+ (!HeapTupleIsHeapOnly(tuple) || warm_update))
+ {
+ if (warm_update)
+ ItemPointerSet(&root_tid,
+ ItemPointerGetBlockNumber(&(tuple->t_self)),
+ HeapTupleHeaderGetRootOffset(tuple->t_data));
+ else
+ {
+ ItemPointerCopy(&tuple->t_self, &root_tid);
+ bms_free(modified_attrs);
+ modified_attrs = NULL;
+ }
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
+ &root_tid,
+ modified_attrs,
estate, false, NULL, NIL);
+ }
}
if (canSetTag)
diff --git b/src/backend/postmaster/pgstat.c a/src/backend/postmaster/pgstat.c
index 3a50488..806d812 100644
--- b/src/backend/postmaster/pgstat.c
+++ a/src/backend/postmaster/pgstat.c
@@ -1824,7 +1824,7 @@ pgstat_count_heap_insert(Relation rel, PgStat_Counter n)
* pgstat_count_heap_update - count a tuple update
*/
void
-pgstat_count_heap_update(Relation rel, bool hot)
+pgstat_count_heap_update(Relation rel, bool hot, bool warm)
{
PgStat_TableStatus *pgstat_info = rel->pgstat_info;
@@ -1842,6 +1842,8 @@ pgstat_count_heap_update(Relation rel, bool hot)
/* t_tuples_hot_updated is nontransactional, so just advance it */
if (hot)
pgstat_info->t_counts.t_tuples_hot_updated++;
+ else if (warm)
+ pgstat_info->t_counts.t_tuples_warm_updated++;
}
}
@@ -4324,6 +4326,7 @@ pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create)
result->tuples_updated = 0;
result->tuples_deleted = 0;
result->tuples_hot_updated = 0;
+ result->tuples_warm_updated = 0;
result->n_live_tuples = 0;
result->n_dead_tuples = 0;
result->changes_since_analyze = 0;
@@ -5433,6 +5436,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len)
tabentry->tuples_updated = tabmsg->t_counts.t_tuples_updated;
tabentry->tuples_deleted = tabmsg->t_counts.t_tuples_deleted;
tabentry->tuples_hot_updated = tabmsg->t_counts.t_tuples_hot_updated;
+ tabentry->tuples_warm_updated = tabmsg->t_counts.t_tuples_warm_updated;
tabentry->n_live_tuples = tabmsg->t_counts.t_delta_live_tuples;
tabentry->n_dead_tuples = tabmsg->t_counts.t_delta_dead_tuples;
tabentry->changes_since_analyze = tabmsg->t_counts.t_changed_tuples;
@@ -5460,6 +5464,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len)
tabentry->tuples_updated += tabmsg->t_counts.t_tuples_updated;
tabentry->tuples_deleted += tabmsg->t_counts.t_tuples_deleted;
tabentry->tuples_hot_updated += tabmsg->t_counts.t_tuples_hot_updated;
+ tabentry->tuples_warm_updated += tabmsg->t_counts.t_tuples_warm_updated;
/* If table was truncated, first reset the live/dead counters */
if (tabmsg->t_counts.t_truncated)
{
diff --git b/src/backend/replication/logical/decode.c a/src/backend/replication/logical/decode.c
index 5c13d26..7a9b48a 100644
--- b/src/backend/replication/logical/decode.c
+++ a/src/backend/replication/logical/decode.c
@@ -347,7 +347,7 @@ DecodeStandbyOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
static void
DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
- uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP_OPMASK;
+ uint8 info = XLogRecGetInfo(buf->record) & XLOG_HEAP2_OPMASK;
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
@@ -359,10 +359,6 @@ DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
switch (info)
{
- case XLOG_HEAP2_MULTI_INSERT:
- if (SnapBuildProcessChange(builder, xid, buf->origptr))
- DecodeMultiInsert(ctx, buf);
- break;
case XLOG_HEAP2_NEW_CID:
{
xl_heap_new_cid *xlrec;
@@ -390,6 +386,7 @@ DecodeHeap2Op(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_HEAP2_CLEANUP_INFO:
case XLOG_HEAP2_VISIBLE:
case XLOG_HEAP2_LOCK_UPDATED:
+ case XLOG_HEAP2_WARMCLEAR:
break;
default:
elog(ERROR, "unexpected RM_HEAP2_ID record type: %u", info);
@@ -418,6 +415,10 @@ DecodeHeapOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (SnapBuildProcessChange(builder, xid, buf->origptr))
DecodeInsert(ctx, buf);
break;
+ case XLOG_HEAP_MULTI_INSERT:
+ if (SnapBuildProcessChange(builder, xid, buf->origptr))
+ DecodeMultiInsert(ctx, buf);
+ break;
/*
* Treat HOT update as normal updates. There is no useful
@@ -809,7 +810,7 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
/*
- * Decode XLOG_HEAP2_MULTI_INSERT_insert record into multiple tuplebufs.
+ * Decode XLOG_HEAP_MULTI_INSERT_insert record into multiple tuplebufs.
*
* Currently MULTI_INSERT will always contain the full tuples.
*/
diff --git b/src/backend/storage/page/bufpage.c a/src/backend/storage/page/bufpage.c
index fdf045a..8d23e92 100644
--- b/src/backend/storage/page/bufpage.c
+++ a/src/backend/storage/page/bufpage.c
@@ -1151,6 +1151,29 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+/*
+ * PageIndexClearWarmTuples
+ *
+ * Clear the given WARM pointers by resetting the flags stored in the TID
+ * field. We assume there is nothing else in the TID flags other than the WARM
+ * information and clearing all flag bits is safe. If that changes, we must
+ * change this routine as well.
+ */
+void
+PageIndexClearWarmTuples(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems)
+{
+ int i;
+ ItemId itemid;
+ IndexTuple itup;
+
+ for (i = 0; i < nclearitems; i++)
+ {
+ itemid = PageGetItemId(page, clearitemnos[i]);
+ itup = (IndexTuple) PageGetItem(page, itemid);
+ ItemPointerClearFlags(&itup->t_tid);
+ }
+}
/*
* Set checksum for a page in shared buffers.
diff --git b/src/backend/utils/adt/pgstatfuncs.c a/src/backend/utils/adt/pgstatfuncs.c
index a987d0d..b8677f3 100644
--- b/src/backend/utils/adt/pgstatfuncs.c
+++ a/src/backend/utils/adt/pgstatfuncs.c
@@ -145,6 +145,22 @@ pg_stat_get_tuples_hot_updated(PG_FUNCTION_ARGS)
Datum
+pg_stat_get_tuples_warm_updated(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ int64 result;
+ PgStat_StatTabEntry *tabentry;
+
+ if ((tabentry = pgstat_fetch_stat_tabentry(relid)) == NULL)
+ result = 0;
+ else
+ result = (int64) (tabentry->tuples_warm_updated);
+
+ PG_RETURN_INT64(result);
+}
+
+
+Datum
pg_stat_get_live_tuples(PG_FUNCTION_ARGS)
{
Oid relid = PG_GETARG_OID(0);
@@ -1644,6 +1660,21 @@ pg_stat_get_xact_tuples_hot_updated(PG_FUNCTION_ARGS)
}
Datum
+pg_stat_get_xact_tuples_warm_updated(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ int64 result;
+ PgStat_TableStatus *tabentry;
+
+ if ((tabentry = find_tabstat_entry(relid)) == NULL)
+ result = 0;
+ else
+ result = (int64) (tabentry->t_counts.t_tuples_warm_updated);
+
+ PG_RETURN_INT64(result);
+}
+
+Datum
pg_stat_get_xact_blocks_fetched(PG_FUNCTION_ARGS)
{
Oid relid = PG_GETARG_OID(0);
diff --git b/src/backend/utils/cache/relcache.c a/src/backend/utils/cache/relcache.c
index ce55fc5..64dbaaa 100644
--- b/src/backend/utils/cache/relcache.c
+++ a/src/backend/utils/cache/relcache.c
@@ -2338,6 +2338,7 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
list_free_deep(relation->rd_fkeylist);
list_free(relation->rd_indexlist);
bms_free(relation->rd_indexattr);
+ bms_free(relation->rd_exprindexattr);
bms_free(relation->rd_keyattr);
bms_free(relation->rd_pkattr);
bms_free(relation->rd_idattr);
@@ -4352,6 +4353,13 @@ RelationGetIndexList(Relation relation)
return list_copy(relation->rd_indexlist);
/*
+ * If the index list was invalidated, we better also invalidate the index
+ * attribute list (which should automatically invalidate other attributes
+ * such as primary key and replica identity)
+ */
+ relation->rd_indexattr = NULL;
+
+ /*
* We build the list we intend to return (in the caller's context) while
* doing the scan. After successfully completing the scan, we copy that
* list into the relcache entry. This avoids cache-context memory leakage
@@ -4759,15 +4767,19 @@ Bitmapset *
RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
{
Bitmapset *indexattrs; /* indexed columns */
+ Bitmapset *exprindexattrs; /* indexed columns in expression/prediacate
+ indexes */
Bitmapset *uindexattrs; /* columns in unique indexes */
Bitmapset *pkindexattrs; /* columns in the primary index */
Bitmapset *idindexattrs; /* columns in the replica identity */
+ Bitmapset *indxnotreadyattrs; /* columns in not ready indexes */
List *indexoidlist;
List *newindexoidlist;
Oid relpkindex;
Oid relreplindex;
ListCell *l;
MemoryContext oldcxt;
+ bool supportswarm = true;/* True if the table can be WARM updated */
/* Quick exit if we already computed the result. */
if (relation->rd_indexattr != NULL)
@@ -4782,6 +4794,10 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
return bms_copy(relation->rd_pkattr);
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
return bms_copy(relation->rd_idattr);
+ case INDEX_ATTR_BITMAP_EXPR_PREDICATE:
+ return bms_copy(relation->rd_exprindexattr);
+ case INDEX_ATTR_BITMAP_NOTREADY:
+ return bms_copy(relation->rd_indxnotreadyattr);
default:
elog(ERROR, "unknown attrKind %u", attrKind);
}
@@ -4822,9 +4838,11 @@ restart:
* won't be returned at all by RelationGetIndexList.
*/
indexattrs = NULL;
+ exprindexattrs = NULL;
uindexattrs = NULL;
pkindexattrs = NULL;
idindexattrs = NULL;
+ indxnotreadyattrs = NULL;
foreach(l, indexoidlist)
{
Oid indexOid = lfirst_oid(l);
@@ -4861,6 +4879,10 @@ restart:
indexattrs = bms_add_member(indexattrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
+ if (!indexInfo->ii_ReadyForInserts)
+ indxnotreadyattrs = bms_add_member(indxnotreadyattrs,
+ attrnum - FirstLowInvalidHeapAttributeNumber);
+
if (isKey)
uindexattrs = bms_add_member(uindexattrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
@@ -4876,10 +4898,29 @@ restart:
}
/* Collect all attributes used in expressions, too */
- pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexattrs);
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &exprindexattrs);
/* Collect all attributes in the index predicate, too */
- pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &indexattrs);
+ pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &exprindexattrs);
+
+ /*
+ * indexattrs should include attributes referenced in index expressions
+ * and predicates too.
+ */
+ indexattrs = bms_add_members(indexattrs, exprindexattrs);
+
+ if (!indexInfo->ii_ReadyForInserts)
+ indxnotreadyattrs = bms_add_members(indxnotreadyattrs,
+ exprindexattrs);
+
+ /*
+ * Check if the index has amrecheck method defined. If the method is
+ * not defined, the index does not support WARM update. Completely
+ * disable WARM updates on such tables.
+ */
+ if (!indexDesc->rd_amroutine->amrecheck)
+ supportswarm = false;
+
index_close(indexDesc, AccessShareLock);
}
@@ -4912,15 +4953,22 @@ restart:
goto restart;
}
+ /* Remember if the table can do WARM updates */
+ relation->rd_supportswarm = supportswarm;
+
/* Don't leak the old values of these bitmaps, if any */
bms_free(relation->rd_indexattr);
relation->rd_indexattr = NULL;
+ bms_free(relation->rd_exprindexattr);
+ relation->rd_exprindexattr = NULL;
bms_free(relation->rd_keyattr);
relation->rd_keyattr = NULL;
bms_free(relation->rd_pkattr);
relation->rd_pkattr = NULL;
bms_free(relation->rd_idattr);
relation->rd_idattr = NULL;
+ bms_free(relation->rd_indxnotreadyattr);
+ relation->rd_indxnotreadyattr = NULL;
/*
* Now save copies of the bitmaps in the relcache entry. We intentionally
@@ -4933,7 +4981,9 @@ restart:
relation->rd_keyattr = bms_copy(uindexattrs);
relation->rd_pkattr = bms_copy(pkindexattrs);
relation->rd_idattr = bms_copy(idindexattrs);
- relation->rd_indexattr = bms_copy(indexattrs);
+ relation->rd_exprindexattr = bms_copy(exprindexattrs);
+ relation->rd_indexattr = bms_copy(bms_union(indexattrs, exprindexattrs));
+ relation->rd_indxnotreadyattr = bms_copy(indxnotreadyattrs);
MemoryContextSwitchTo(oldcxt);
/* We return our original working copy for caller to play with */
@@ -4947,6 +4997,10 @@ restart:
return bms_copy(relation->rd_pkattr);
case INDEX_ATTR_BITMAP_IDENTITY_KEY:
return idindexattrs;
+ case INDEX_ATTR_BITMAP_EXPR_PREDICATE:
+ return exprindexattrs;
+ case INDEX_ATTR_BITMAP_NOTREADY:
+ return indxnotreadyattrs;
default:
elog(ERROR, "unknown attrKind %u", attrKind);
return NULL;
@@ -5559,6 +5613,7 @@ load_relcache_init_file(bool shared)
rel->rd_keyattr = NULL;
rel->rd_pkattr = NULL;
rel->rd_idattr = NULL;
+ rel->rd_indxnotreadyattr = NULL;
rel->rd_pubactions = NULL;
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
diff --git b/src/backend/utils/time/combocid.c a/src/backend/utils/time/combocid.c
index baff998..6a2e2f2 100644
--- b/src/backend/utils/time/combocid.c
+++ a/src/backend/utils/time/combocid.c
@@ -106,7 +106,7 @@ HeapTupleHeaderGetCmin(HeapTupleHeader tup)
{
CommandId cid = HeapTupleHeaderGetRawCommandId(tup);
- Assert(!(tup->t_infomask & HEAP_MOVED));
+ Assert(!(HeapTupleHeaderIsMoved(tup)));
Assert(TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tup)));
if (tup->t_infomask & HEAP_COMBOCID)
@@ -120,7 +120,7 @@ HeapTupleHeaderGetCmax(HeapTupleHeader tup)
{
CommandId cid = HeapTupleHeaderGetRawCommandId(tup);
- Assert(!(tup->t_infomask & HEAP_MOVED));
+ Assert(!(HeapTupleHeaderIsMoved(tup)));
/*
* Because GetUpdateXid() performs memory allocations if xmax is a
diff --git b/src/backend/utils/time/tqual.c a/src/backend/utils/time/tqual.c
index 519f3b6..e54d0df 100644
--- b/src/backend/utils/time/tqual.c
+++ a/src/backend/utils/time/tqual.c
@@ -186,7 +186,7 @@ HeapTupleSatisfiesSelf(HeapTuple htup, Snapshot snapshot, Buffer buffer)
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -205,7 +205,7 @@ HeapTupleSatisfiesSelf(HeapTuple htup, Snapshot snapshot, Buffer buffer)
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -377,7 +377,7 @@ HeapTupleSatisfiesToast(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -396,7 +396,7 @@ HeapTupleSatisfiesToast(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -471,7 +471,7 @@ HeapTupleSatisfiesUpdate(HeapTuple htup, CommandId curcid,
return HeapTupleInvisible;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -490,7 +490,7 @@ HeapTupleSatisfiesUpdate(HeapTuple htup, CommandId curcid,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -753,7 +753,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -772,7 +772,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -974,7 +974,7 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
return false;
/* Used by pre-9.0 binary upgrades */
- if (tuple->t_infomask & HEAP_MOVED_OFF)
+ if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -993,7 +993,7 @@ HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
}
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -1180,7 +1180,7 @@ HeapTupleSatisfiesVacuum(HeapTuple htup, TransactionId OldestXmin,
if (HeapTupleHeaderXminInvalid(tuple))
return HEAPTUPLE_DEAD;
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_OFF)
+ else if (HeapTupleHeaderIsMovedOff(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
@@ -1198,7 +1198,7 @@ HeapTupleSatisfiesVacuum(HeapTuple htup, TransactionId OldestXmin,
InvalidTransactionId);
}
/* Used by pre-9.0 binary upgrades */
- else if (tuple->t_infomask & HEAP_MOVED_IN)
+ else if (HeapTupleHeaderIsMovedIn(tuple))
{
TransactionId xvac = HeapTupleHeaderGetXvac(tuple);
diff --git b/src/include/access/amapi.h a/src/include/access/amapi.h
index f919cf8..8b7af1e 100644
--- b/src/include/access/amapi.h
+++ a/src/include/access/amapi.h
@@ -13,6 +13,7 @@
#define AMAPI_H
#include "access/genam.h"
+#include "access/itup.h"
/*
* We don't wish to include planner header files here, since most of an index
@@ -74,6 +75,14 @@ typedef bool (*aminsert_function) (Relation indexRelation,
Relation heapRelation,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+/* insert this WARM tuple */
+typedef bool (*amwarminsert_function) (Relation indexRelation,
+ Datum *values,
+ bool *isnull,
+ ItemPointer heap_tid,
+ Relation heapRelation,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
/* bulk delete */
typedef IndexBulkDeleteResult *(*ambulkdelete_function) (IndexVacuumInfo *info,
@@ -152,6 +161,11 @@ typedef void (*aminitparallelscan_function) (void *target);
/* (re)start parallel index scan */
typedef void (*amparallelrescan_function) (IndexScanDesc scan);
+/* recheck index tuple and heap tuple match */
+typedef bool (*amrecheck_function) (Relation indexRel,
+ struct IndexInfo *indexInfo, IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple);
+
/*
* API struct for an index AM. Note this must be stored in a single palloc'd
* chunk of memory.
@@ -198,6 +212,7 @@ typedef struct IndexAmRoutine
ambuild_function ambuild;
ambuildempty_function ambuildempty;
aminsert_function aminsert;
+ amwarminsert_function amwarminsert;
ambulkdelete_function ambulkdelete;
amvacuumcleanup_function amvacuumcleanup;
amcanreturn_function amcanreturn; /* can be NULL */
@@ -217,6 +232,9 @@ typedef struct IndexAmRoutine
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
aminitparallelscan_function aminitparallelscan; /* can be NULL */
amparallelrescan_function amparallelrescan; /* can be NULL */
+
+ /* interface function to support WARM */
+ amrecheck_function amrecheck; /* can be NULL */
} IndexAmRoutine;
diff --git b/src/include/access/genam.h a/src/include/access/genam.h
index f467b18..965be45 100644
--- b/src/include/access/genam.h
+++ a/src/include/access/genam.h
@@ -75,12 +75,29 @@ typedef struct IndexBulkDeleteResult
bool estimated_count; /* num_index_tuples is an estimate */
double num_index_tuples; /* tuples remaining */
double tuples_removed; /* # removed during vacuum operation */
+ double num_warm_pointers; /* # WARM pointers found */
+ double num_clear_pointers; /* # CLEAR pointers found */
+ double pointers_cleared; /* # WARM pointers cleared */
+ double warm_pointers_removed; /* # WARM pointers removed */
+ double clear_pointers_removed; /* # CLEAR pointers removed */
BlockNumber pages_deleted; /* # unused pages in index */
BlockNumber pages_free; /* # pages available for reuse */
} IndexBulkDeleteResult;
+/*
+ * IndexBulkDeleteCallback should return one of the following
+ */
+typedef enum IndexBulkDeleteCallbackResult
+{
+ IBDCR_KEEP, /* index tuple should be preserved */
+ IBDCR_DELETE, /* index tuple should be deleted */
+ IBDCR_CLEAR_WARM /* index tuple should be cleared of WARM bit */
+} IndexBulkDeleteCallbackResult;
+
/* Typedef for callback function to determine if a tuple is bulk-deletable */
-typedef bool (*IndexBulkDeleteCallback) (ItemPointer itemptr, void *state);
+typedef IndexBulkDeleteCallbackResult (*IndexBulkDeleteCallback) (
+ ItemPointer itemptr,
+ bool is_warm, void *state);
/* struct definitions appear in relscan.h */
typedef struct IndexScanDescData *IndexScanDesc;
@@ -135,7 +152,8 @@ extern bool index_insert(Relation indexRelation,
ItemPointer heap_t_ctid,
Relation heapRelation,
IndexUniqueCheck checkUnique,
- struct IndexInfo *indexInfo);
+ struct IndexInfo *indexInfo,
+ bool warm_update);
extern IndexScanDesc index_beginscan(Relation heapRelation,
Relation indexRelation,
diff --git b/src/include/access/hash.h a/src/include/access/hash.h
index eb1df57..12a4bb6 100644
--- b/src/include/access/hash.h
+++ a/src/include/access/hash.h
@@ -281,6 +281,11 @@ typedef HashMetaPageData *HashMetaPage;
#define HASHPROC 1
#define HASHNProcs 1
+/*
+ * Flags overloaded on t_tid.ip_posid field. They are managed by
+ * ItemPointerSetFlags and corresponing routines.
+ */
+#define HASH_INDEX_WARM_POINTER 0x01
/* public routines */
@@ -291,6 +296,10 @@ extern bool hashinsert(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+extern bool hashwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
extern bool hashgettuple(IndexScanDesc scan, ScanDirection dir);
extern int64 hashgetbitmap(IndexScanDesc scan, TIDBitmap *tbm);
extern IndexScanDesc hashbeginscan(Relation rel, int nkeys, int norderbys);
@@ -360,6 +369,8 @@ extern void _hash_expandtable(Relation rel, Buffer metabuf);
extern void _hash_finish_split(Relation rel, Buffer metabuf, Buffer obuf,
Bucket obucket, uint32 maxbucket, uint32 highmask,
uint32 lowmask);
+extern void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems);
/* hashsearch.c */
extern bool _hash_next(IndexScanDesc scan, ScanDirection dir);
@@ -401,7 +412,14 @@ extern void hashbucketcleanup(Relation rel, Bucket cur_bucket,
BufferAccessStrategy bstrategy,
uint32 maxbucket, uint32 highmask, uint32 lowmask,
double *tuples_removed, double *num_index_tuples,
+ double *warm_pointers_removed,
+ double *clear_pointers_removed,
+ double *pointers_cleared,
bool bucket_has_garbage,
IndexBulkDeleteCallback callback, void *callback_state);
+/* hash.c */
+extern bool hashrecheck(Relation indexRel, struct IndexInfo *indexInfo,
+ IndexTuple indexTuple, Relation heapRel, HeapTuple heapTuple);
+
#endif /* HASH_H */
diff --git b/src/include/access/hash_xlog.h a/src/include/access/hash_xlog.h
index dfd9237..0549a5a 100644
--- b/src/include/access/hash_xlog.h
+++ a/src/include/access/hash_xlog.h
@@ -199,9 +199,10 @@ typedef struct xl_hash_delete
{
bool is_primary_bucket_page; /* TRUE if the operation is for
* primary bucket page */
+ uint16 nclearitems; /* # of items to clear of WARM bits */
} xl_hash_delete;
-#define SizeOfHashDelete (offsetof(xl_hash_delete, is_primary_bucket_page) + sizeof(bool))
+#define SizeOfHashDelete (offsetof(xl_hash_delete, nclearitems) + sizeof(uint16))
/*
* This is what we need for metapage update operation.
diff --git b/src/include/access/heapam.h a/src/include/access/heapam.h
index 5540e12..2217af9 100644
--- b/src/include/access/heapam.h
+++ a/src/include/access/heapam.h
@@ -72,6 +72,20 @@ typedef struct HeapUpdateFailureData
CommandId cmax;
} HeapUpdateFailureData;
+typedef int HeapCheckWarmChainStatus;
+
+#define HCWC_CLEAR_TUPLE 0x0001
+#define HCWC_WARM_TUPLE 0x0002
+#define HCWC_WARM_UPDATED_TUPLE 0x0004
+
+#define HCWC_IS_MIXED(status) \
+ (((status) & (HCWC_CLEAR_TUPLE | HCWC_WARM_TUPLE)) != 0)
+#define HCWC_IS_ALL_WARM(status) \
+ (((status) & HCWC_CLEAR_TUPLE) == 0)
+#define HCWC_IS_ALL_CLEAR(status) \
+ (((status) & HCWC_WARM_TUPLE) == 0)
+#define HCWC_IS_WARM_UPDATED(status) \
+ (((status) & HCWC_WARM_UPDATED_TUPLE) != 0)
/* ----------------
* function prototypes for heap access method
@@ -137,9 +151,10 @@ extern bool heap_fetch(Relation relation, Snapshot snapshot,
Relation stats_relation);
extern bool heap_hot_search_buffer(ItemPointer tid, Relation relation,
Buffer buffer, Snapshot snapshot, HeapTuple heapTuple,
- bool *all_dead, bool first_call);
+ bool *all_dead, bool first_call, bool *recheck);
extern bool heap_hot_search(ItemPointer tid, Relation relation,
- Snapshot snapshot, bool *all_dead);
+ Snapshot snapshot, bool *all_dead,
+ bool *recheck, Buffer *buffer, HeapTuple heapTuple);
extern void heap_get_latest_tid(Relation relation, Snapshot snapshot,
ItemPointer tid);
@@ -161,7 +176,8 @@ extern void heap_abort_speculative(Relation relation, HeapTuple tuple);
extern HTSU_Result heap_update(Relation relation, ItemPointer otid,
HeapTuple newtup,
CommandId cid, Snapshot crosscheck, bool wait,
- HeapUpdateFailureData *hufd, LockTupleMode *lockmode);
+ HeapUpdateFailureData *hufd, LockTupleMode *lockmode,
+ Bitmapset **modified_attrsp, bool *warm_update);
extern HTSU_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy,
bool follow_update,
@@ -176,10 +192,16 @@ extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
extern Oid simple_heap_insert(Relation relation, HeapTuple tup);
extern void simple_heap_delete(Relation relation, ItemPointer tid);
extern void simple_heap_update(Relation relation, ItemPointer otid,
- HeapTuple tup);
+ HeapTuple tup,
+ Bitmapset **modified_attrs,
+ bool *warm_update);
extern void heap_sync(Relation relation);
extern void heap_update_snapshot(HeapScanDesc scan, Snapshot snapshot);
+extern HeapCheckWarmChainStatus heap_check_warm_chain(Page dp,
+ ItemPointer tid, bool stop_at_warm);
+extern int heap_clear_warm_chain(Page dp, ItemPointer tid,
+ OffsetNumber *cleared_offnums);
/* in heap/pruneheap.c */
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
diff --git b/src/include/access/heapam_xlog.h a/src/include/access/heapam_xlog.h
index e6019d5..66fd0ea 100644
--- b/src/include/access/heapam_xlog.h
+++ a/src/include/access/heapam_xlog.h
@@ -32,7 +32,7 @@
#define XLOG_HEAP_INSERT 0x00
#define XLOG_HEAP_DELETE 0x10
#define XLOG_HEAP_UPDATE 0x20
-/* 0x030 is free, was XLOG_HEAP_MOVE */
+#define XLOG_HEAP_MULTI_INSERT 0x30
#define XLOG_HEAP_HOT_UPDATE 0x40
#define XLOG_HEAP_CONFIRM 0x50
#define XLOG_HEAP_LOCK 0x60
@@ -47,18 +47,23 @@
/*
* We ran out of opcodes, so heapam.c now has a second RmgrId. These opcodes
* are associated with RM_HEAP2_ID, but are not logically different from
- * the ones above associated with RM_HEAP_ID. XLOG_HEAP_OPMASK applies to
- * these, too.
+ * the ones above associated with RM_HEAP_ID.
+ *
+ * In PG 10, we moved XLOG_HEAP2_MULTI_INSERT to RM_HEAP_ID. That allows us to
+ * use 0x80 bit in RM_HEAP2_ID, thus potentially creating another 8 possible
+ * opcodes in RM_HEAP2_ID.
*/
#define XLOG_HEAP2_REWRITE 0x00
#define XLOG_HEAP2_CLEAN 0x10
#define XLOG_HEAP2_FREEZE_PAGE 0x20
#define XLOG_HEAP2_CLEANUP_INFO 0x30
#define XLOG_HEAP2_VISIBLE 0x40
-#define XLOG_HEAP2_MULTI_INSERT 0x50
+#define XLOG_HEAP2_WARMCLEAR 0x50
#define XLOG_HEAP2_LOCK_UPDATED 0x60
#define XLOG_HEAP2_NEW_CID 0x70
+#define XLOG_HEAP2_OPMASK 0x70
+
/*
* xl_heap_insert/xl_heap_multi_insert flag values, 8 bits are available.
*/
@@ -80,6 +85,7 @@
#define XLH_UPDATE_CONTAINS_NEW_TUPLE (1<<4)
#define XLH_UPDATE_PREFIX_FROM_OLD (1<<5)
#define XLH_UPDATE_SUFFIX_FROM_OLD (1<<6)
+#define XLH_UPDATE_WARM_UPDATE (1<<7)
/* convenience macro for checking whether any form of old tuple was logged */
#define XLH_UPDATE_CONTAINS_OLD \
@@ -225,6 +231,14 @@ typedef struct xl_heap_clean
#define SizeOfHeapClean (offsetof(xl_heap_clean, ndead) + sizeof(uint16))
+typedef struct xl_heap_warmclear
+{
+ uint16 ncleared;
+ /* OFFSET NUMBERS are in the block reference 0 */
+} xl_heap_warmclear;
+
+#define SizeOfHeapWarmClear (offsetof(xl_heap_warmclear, ncleared) + sizeof(uint16))
+
/*
* Cleanup_info is required in some cases during a lazy VACUUM.
* Used for reporting the results of HeapTupleHeaderAdvanceLatestRemovedXid()
@@ -388,6 +402,8 @@ extern XLogRecPtr log_heap_clean(Relation reln, Buffer buffer,
OffsetNumber *nowdead, int ndead,
OffsetNumber *nowunused, int nunused,
TransactionId latestRemovedXid);
+extern XLogRecPtr log_heap_warmclear(Relation reln, Buffer buffer,
+ OffsetNumber *cleared, int ncleared);
extern XLogRecPtr log_heap_freeze(Relation reln, Buffer buffer,
TransactionId cutoff_xid, xl_heap_freeze_tuple *tuples,
int ntuples);
diff --git b/src/include/access/htup_details.h a/src/include/access/htup_details.h
index 4d614b7..bcefba6 100644
--- b/src/include/access/htup_details.h
+++ a/src/include/access/htup_details.h
@@ -201,6 +201,21 @@ struct HeapTupleHeaderData
* upgrade support */
#define HEAP_MOVED (HEAP_MOVED_OFF | HEAP_MOVED_IN)
+/*
+ * A WARM chain usually consists of two parts. Each of these parts are HOT
+ * chains in themselves i.e. all indexed columns has the same value, but a WARM
+ * update separates these parts. We need a mechanism to identify which part a
+ * tuple belongs to. We can't just look at if it's a
+ * HeapTupleHeaderIsWarmUpdated() because during WARM update, both old and new
+ * tuples are marked as WARM tuples.
+ *
+ * We need another infomask bit for this. But we use the same infomask bit that
+ * was earlier used for by old-style VACUUM FULL. This is safe because
+ * HEAP_WARM_TUPLE flag will always be set along with HEAP_WARM_UPDATED. So if
+ * HEAP_WARM_TUPLE and HEAP_WARM_UPDATED is set then we know that it's
+ * referring to red part of the WARM chain.
+ */
+#define HEAP_WARM_TUPLE 0x4000
#define HEAP_XACT_MASK 0xFFF0 /* visibility-related bits */
/*
@@ -260,7 +275,11 @@ struct HeapTupleHeaderData
* information stored in t_infomask2:
*/
#define HEAP_NATTS_MASK 0x07FF /* 11 bits for number of attributes */
-/* bits 0x0800 are available */
+#define HEAP_WARM_UPDATED 0x0800 /*
+ * This or a prior version of this
+ * tuple in the current HOT chain was
+ * once WARM updated
+ */
#define HEAP_LATEST_TUPLE 0x1000 /*
* This is the last tuple in chain and
* ip_posid points to the root line
@@ -271,7 +290,7 @@ struct HeapTupleHeaderData
#define HEAP_HOT_UPDATED 0x4000 /* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE 0x8000 /* this is heap-only tuple */
-#define HEAP2_XACT_MASK 0xF000 /* visibility-related bits */
+#define HEAP2_XACT_MASK 0xF800 /* visibility-related bits */
/*
@@ -396,7 +415,7 @@ struct HeapTupleHeaderData
/* SetCmin is reasonably simple since we never need a combo CID */
#define HeapTupleHeaderSetCmin(tup, cid) \
do { \
- Assert(!((tup)->t_infomask & HEAP_MOVED)); \
+ Assert(!HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
(tup)->t_infomask &= ~HEAP_COMBOCID; \
} while (0)
@@ -404,7 +423,7 @@ do { \
/* SetCmax must be used after HeapTupleHeaderAdjustCmax; see combocid.c */
#define HeapTupleHeaderSetCmax(tup, cid, iscombo) \
do { \
- Assert(!((tup)->t_infomask & HEAP_MOVED)); \
+ Assert(!HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
if (iscombo) \
(tup)->t_infomask |= HEAP_COMBOCID; \
@@ -414,7 +433,7 @@ do { \
#define HeapTupleHeaderGetXvac(tup) \
( \
- ((tup)->t_infomask & HEAP_MOVED) ? \
+ HeapTupleHeaderIsMoved(tup) ? \
(tup)->t_choice.t_heap.t_field3.t_xvac \
: \
InvalidTransactionId \
@@ -422,7 +441,7 @@ do { \
#define HeapTupleHeaderSetXvac(tup, xid) \
do { \
- Assert((tup)->t_infomask & HEAP_MOVED); \
+ Assert(HeapTupleHeaderIsMoved(tup)); \
(tup)->t_choice.t_heap.t_field3.t_xvac = (xid); \
} while (0)
@@ -510,6 +529,21 @@ do { \
((tup)->t_infomask2 & HEAP_ONLY_TUPLE) != 0 \
)
+#define HeapTupleHeaderSetWarmUpdated(tup) \
+do { \
+ (tup)->t_infomask2 |= HEAP_WARM_UPDATED; \
+} while (0)
+
+#define HeapTupleHeaderClearWarmUpdated(tup) \
+do { \
+ (tup)->t_infomask2 &= ~HEAP_WARM_UPDATED; \
+} while (0)
+
+#define HeapTupleHeaderIsWarmUpdated(tup) \
+( \
+ ((tup)->t_infomask2 & HEAP_WARM_UPDATED) != 0 \
+)
+
/*
* Mark this as the last tuple in the HOT chain. Before PG v10 we used to store
* the TID of the tuple itself in t_ctid field to mark the end of the chain.
@@ -635,6 +669,58 @@ do { \
)
/*
+ * Macros to check if tuple is a moved-off/in tuple by VACUUM FULL in from
+ * pre-9.0 era. Such tuple must not have HEAP_WARM_TUPLE flag set.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderIsMovedOff(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED_OFF) \
+)
+
+#define HeapTupleHeaderIsMovedIn(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED_IN) \
+)
+
+#define HeapTupleHeaderIsMoved(tuple) \
+( \
+ !HeapTupleHeaderIsWarmUpdated((tuple)) && \
+ ((tuple)->t_infomask & HEAP_MOVED) \
+)
+
+/*
+ * Check if tuple belongs to the second part of the WARM chain.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderIsWarm(tuple) \
+( \
+ HeapTupleHeaderIsWarmUpdated(tuple) && \
+ (((tuple)->t_infomask & HEAP_WARM_TUPLE) != 0) \
+)
+
+/*
+ * Mark tuple as a member of the second part of the chain. Must only be done on
+ * a tuple which is already marked a WARM-tuple.
+ *
+ * Beware of multiple evaluations of the argument.
+ */
+#define HeapTupleHeaderSetWarm(tuple) \
+( \
+ AssertMacro(HeapTupleHeaderIsWarmUpdated(tuple)), \
+ (tuple)->t_infomask |= HEAP_WARM_TUPLE \
+)
+
+#define HeapTupleHeaderClearWarm(tuple) \
+( \
+ (tuple)->t_infomask &= ~HEAP_WARM_TUPLE \
+)
+
+/*
* BITMAPLEN(NATTS) -
* Computes size of null bitmap given number of data columns.
*/
@@ -785,6 +871,24 @@ struct MinimalTupleData
#define HeapTupleClearHeapOnly(tuple) \
HeapTupleHeaderClearHeapOnly((tuple)->t_data)
+#define HeapTupleIsWarmUpdated(tuple) \
+ HeapTupleHeaderIsWarmUpdated((tuple)->t_data)
+
+#define HeapTupleSetWarmUpdated(tuple) \
+ HeapTupleHeaderSetWarmUpdated((tuple)->t_data)
+
+#define HeapTupleClearWarmUpdated(tuple) \
+ HeapTupleHeaderClearWarmUpdated((tuple)->t_data)
+
+#define HeapTupleIsWarm(tuple) \
+ HeapTupleHeaderIsWarm((tuple)->t_data)
+
+#define HeapTupleSetWarm(tuple) \
+ HeapTupleHeaderSetWarm((tuple)->t_data)
+
+#define HeapTupleClearWarm(tuple) \
+ HeapTupleHeaderClearWarm((tuple)->t_data)
+
#define HeapTupleGetOid(tuple) \
HeapTupleHeaderGetOid((tuple)->t_data)
diff --git b/src/include/access/nbtree.h a/src/include/access/nbtree.h
index f9304db..163180d 100644
--- b/src/include/access/nbtree.h
+++ a/src/include/access/nbtree.h
@@ -427,6 +427,12 @@ typedef BTScanOpaqueData *BTScanOpaque;
#define SK_BT_NULLS_FIRST (INDOPTION_NULLS_FIRST << SK_BT_INDOPTION_SHIFT)
/*
+ * Flags overloaded on t_tid.ip_posid field. They are managed by
+ * ItemPointerSetFlags and corresponing routines.
+ */
+#define BTREE_INDEX_WARM_POINTER 0x01
+
+/*
* external entry points for btree, in nbtree.c
*/
extern IndexBuildResult *btbuild(Relation heap, Relation index,
@@ -436,6 +442,10 @@ extern bool btinsert(Relation rel, Datum *values, bool *isnull,
ItemPointer ht_ctid, Relation heapRel,
IndexUniqueCheck checkUnique,
struct IndexInfo *indexInfo);
+extern bool btwarminsert(Relation rel, Datum *values, bool *isnull,
+ ItemPointer ht_ctid, Relation heapRel,
+ IndexUniqueCheck checkUnique,
+ struct IndexInfo *indexInfo);
extern IndexScanDesc btbeginscan(Relation rel, int nkeys, int norderbys);
extern Size btestimateparallelscan(void);
extern void btinitparallelscan(void *target);
@@ -487,10 +497,12 @@ extern void _bt_pageinit(Page page, Size size);
extern bool _bt_page_recyclable(Page page);
extern void _bt_delitems_delete(Relation rel, Buffer buf,
OffsetNumber *itemnos, int nitems, Relation heapRel);
-extern void _bt_delitems_vacuum(Relation rel, Buffer buf,
- OffsetNumber *itemnos, int nitems,
- BlockNumber lastBlockVacuumed);
+extern void _bt_handleitems_vacuum(Relation rel, Buffer buf,
+ OffsetNumber *delitemnos, int ndelitems,
+ OffsetNumber *clearitemnos, int nclearitems);
extern int _bt_pagedel(Relation rel, Buffer buf);
+extern void _bt_clear_items(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems);
/*
* prototypes for functions in nbtsearch.c
@@ -537,6 +549,9 @@ extern bytea *btoptions(Datum reloptions, bool validate);
extern bool btproperty(Oid index_oid, int attno,
IndexAMProperty prop, const char *propname,
bool *res, bool *isnull);
+extern bool btrecheck(Relation indexRel, struct IndexInfo *indexInfo,
+ IndexTuple indexTuple,
+ Relation heapRel, HeapTuple heapTuple);
/*
* prototypes for functions in nbtvalidate.c
diff --git b/src/include/access/nbtxlog.h a/src/include/access/nbtxlog.h
index d6a3085..7efd0d7 100644
--- b/src/include/access/nbtxlog.h
+++ a/src/include/access/nbtxlog.h
@@ -142,34 +142,20 @@ typedef struct xl_btree_reuse_page
/*
* This is what we need to know about vacuum of individual leaf index tuples.
* The WAL record can represent deletion of any number of index tuples on a
- * single index page when executed by VACUUM.
- *
- * For MVCC scans, lastBlockVacuumed will be set to InvalidBlockNumber.
- * For a non-MVCC index scans there is an additional correctness requirement
- * for applying these changes during recovery, which is that we must do one
- * of these two things for every block in the index:
- * * lock the block for cleanup and apply any required changes
- * * EnsureBlockUnpinned()
- * The purpose of this is to ensure that no index scans started before we
- * finish scanning the index are still running by the time we begin to remove
- * heap tuples.
- *
- * Any changes to any one block are registered on just one WAL record. All
- * blocks that we need to run EnsureBlockUnpinned() are listed as a block range
- * starting from the last block vacuumed through until this one. Individual
- * block numbers aren't given.
+ * single index page when executed by VACUUM. It also includes tuples which
+ * are cleared of WARM bits by VACUUM.
*
* Note that the *last* WAL record in any vacuum of an index is allowed to
* have a zero length array of offsets. Earlier records must have at least one.
*/
typedef struct xl_btree_vacuum
{
- BlockNumber lastBlockVacuumed;
-
- /* TARGET OFFSET NUMBERS FOLLOW */
+ uint16 ndelitems;
+ uint16 nclearitems;
+ /* ndelitems + nclearitems TARGET OFFSET NUMBERS FOLLOW */
} xl_btree_vacuum;
-#define SizeOfBtreeVacuum (offsetof(xl_btree_vacuum, lastBlockVacuumed) + sizeof(BlockNumber))
+#define SizeOfBtreeVacuum (offsetof(xl_btree_vacuum, nclearitems) + sizeof(uint16))
/*
* This is what we need to know about marking an empty branch for deletion.
diff --git b/src/include/access/relscan.h a/src/include/access/relscan.h
index 3fc726d..fa178d3 100644
--- b/src/include/access/relscan.h
+++ a/src/include/access/relscan.h
@@ -104,6 +104,9 @@ typedef struct IndexScanDescData
/* index access method's private state */
void *opaque; /* access-method-specific info */
+ /* IndexInfo structure for this index */
+ struct IndexInfo *indexInfo;
+
/*
* In an index-only scan, a successful amgettuple call must fill either
* xs_itup (and xs_itupdesc) or xs_hitup (and xs_hitupdesc) to provide the
@@ -119,7 +122,7 @@ typedef struct IndexScanDescData
HeapTupleData xs_ctup; /* current heap tuple, if any */
Buffer xs_cbuf; /* current heap buffer in scan, if any */
/* NB: if xs_cbuf is not InvalidBuffer, we hold a pin on that buffer */
- bool xs_recheck; /* T means scan keys must be rechecked */
+ bool xs_recheck; /* T means scan keys must be rechecked for each tuple */
/*
* When fetching with an ordering operator, the values of the ORDER BY
diff --git b/src/include/catalog/index.h a/src/include/catalog/index.h
index 20bec90..f92ec29 100644
--- b/src/include/catalog/index.h
+++ a/src/include/catalog/index.h
@@ -89,6 +89,13 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
Datum *values,
bool *isnull);
+extern void FormIndexPlainDatum(IndexInfo *indexInfo,
+ Relation heapRel,
+ HeapTuple heapTup,
+ Datum *values,
+ bool *isnull,
+ bool *isavail);
+
extern void index_build(Relation heapRelation,
Relation indexRelation,
IndexInfo *indexInfo,
diff --git b/src/include/catalog/pg_proc.h a/src/include/catalog/pg_proc.h
index 836d6ff..0ca6e22 100644
--- b/src/include/catalog/pg_proc.h
+++ a/src/include/catalog/pg_proc.h
@@ -2769,6 +2769,8 @@ DATA(insert OID = 1933 ( pg_stat_get_tuples_deleted PGNSP PGUID 12 1 0 0 0 f f
DESCR("statistics: number of tuples deleted");
DATA(insert OID = 1972 ( pg_stat_get_tuples_hot_updated PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_hot_updated _null_ _null_ _null_ ));
DESCR("statistics: number of tuples hot updated");
+DATA(insert OID = 3355 ( pg_stat_get_tuples_warm_updated PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_warm_updated _null_ _null_ _null_ ));
+DESCR("statistics: number of tuples warm updated");
DATA(insert OID = 2878 ( pg_stat_get_live_tuples PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_live_tuples _null_ _null_ _null_ ));
DESCR("statistics: number of live tuples");
DATA(insert OID = 2879 ( pg_stat_get_dead_tuples PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_dead_tuples _null_ _null_ _null_ ));
@@ -2921,6 +2923,8 @@ DATA(insert OID = 3042 ( pg_stat_get_xact_tuples_deleted PGNSP PGUID 12 1 0 0
DESCR("statistics: number of tuples deleted in current transaction");
DATA(insert OID = 3043 ( pg_stat_get_xact_tuples_hot_updated PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_tuples_hot_updated _null_ _null_ _null_ ));
DESCR("statistics: number of tuples hot updated in current transaction");
+DATA(insert OID = 3356 ( pg_stat_get_xact_tuples_warm_updated PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_tuples_warm_updated _null_ _null_ _null_ ));
+DESCR("statistics: number of tuples warm updated in current transaction");
DATA(insert OID = 3044 ( pg_stat_get_xact_blocks_fetched PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_blocks_fetched _null_ _null_ _null_ ));
DESCR("statistics: number of blocks fetched in current transaction");
DATA(insert OID = 3045 ( pg_stat_get_xact_blocks_hit PGNSP PGUID 12 1 0 0 0 f f f f t f v r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_xact_blocks_hit _null_ _null_ _null_ ));
diff --git b/src/include/commands/progress.h a/src/include/commands/progress.h
index 9472ecc..b355b61 100644
--- b/src/include/commands/progress.h
+++ a/src/include/commands/progress.h
@@ -25,6 +25,7 @@
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_HEAP_BLKS_WARMCLEARED 7
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git b/src/include/executor/executor.h a/src/include/executor/executor.h
index 02dbe7b..c4495a3 100644
--- b/src/include/executor/executor.h
+++ a/src/include/executor/executor.h
@@ -382,6 +382,7 @@ extern void UnregisterExprContextCallback(ExprContext *econtext,
extern void ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative);
extern void ExecCloseIndices(ResultRelInfo *resultRelInfo);
extern List *ExecInsertIndexTuples(TupleTableSlot *slot, ItemPointer tupleid,
+ ItemPointer root_tid, Bitmapset *modified_attrs,
EState *estate, bool noDupErr, bool *specConflict,
List *arbiterIndexes);
extern bool ExecCheckIndexConstraints(TupleTableSlot *slot, EState *estate,
diff --git b/src/include/executor/nodeIndexscan.h a/src/include/executor/nodeIndexscan.h
index ea3f3a5..ebeec74 100644
--- b/src/include/executor/nodeIndexscan.h
+++ a/src/include/executor/nodeIndexscan.h
@@ -41,5 +41,4 @@ extern void ExecIndexEvalRuntimeKeys(ExprContext *econtext,
extern bool ExecIndexEvalArrayKeys(ExprContext *econtext,
IndexArrayKeyInfo *arrayKeys, int numArrayKeys);
extern bool ExecIndexAdvanceArrayKeys(IndexArrayKeyInfo *arrayKeys, int numArrayKeys);
-
#endif /* NODEINDEXSCAN_H */
diff --git b/src/include/nodes/execnodes.h a/src/include/nodes/execnodes.h
index f856f60..cd09553 100644
--- b/src/include/nodes/execnodes.h
+++ a/src/include/nodes/execnodes.h
@@ -66,6 +66,7 @@ typedef struct IndexInfo
NodeTag type;
int ii_NumIndexAttrs;
AttrNumber ii_KeyAttrNumbers[INDEX_MAX_KEYS];
+ Bitmapset *ii_indxattrs; /* bitmap of all columns used in this index */
List *ii_Expressions; /* list of Expr */
List *ii_ExpressionsState; /* list of ExprState */
List *ii_Predicate; /* list of Expr */
diff --git b/src/include/pgstat.h a/src/include/pgstat.h
index f2daf32..af8a3ba 100644
--- b/src/include/pgstat.h
+++ a/src/include/pgstat.h
@@ -105,6 +105,7 @@ typedef struct PgStat_TableCounts
PgStat_Counter t_tuples_updated;
PgStat_Counter t_tuples_deleted;
PgStat_Counter t_tuples_hot_updated;
+ PgStat_Counter t_tuples_warm_updated;
bool t_truncated;
PgStat_Counter t_delta_live_tuples;
@@ -625,6 +626,7 @@ typedef struct PgStat_StatTabEntry
PgStat_Counter tuples_updated;
PgStat_Counter tuples_deleted;
PgStat_Counter tuples_hot_updated;
+ PgStat_Counter tuples_warm_updated;
PgStat_Counter n_live_tuples;
PgStat_Counter n_dead_tuples;
@@ -1257,7 +1259,7 @@ pgstat_report_wait_end(void)
(pgStatBlockWriteTime += (n))
extern void pgstat_count_heap_insert(Relation rel, PgStat_Counter n);
-extern void pgstat_count_heap_update(Relation rel, bool hot);
+extern void pgstat_count_heap_update(Relation rel, bool hot, bool warm);
extern void pgstat_count_heap_delete(Relation rel);
extern void pgstat_count_truncate(Relation rel);
extern void pgstat_update_heap_dead_tuples(Relation rel, int delta);
diff --git b/src/include/storage/bufpage.h a/src/include/storage/bufpage.h
index e956dc3..1852195 100644
--- b/src/include/storage/bufpage.h
+++ a/src/include/storage/bufpage.h
@@ -433,6 +433,8 @@ extern void PageIndexMultiDelete(Page page, OffsetNumber *itemnos, int nitems);
extern void PageIndexTupleDeleteNoCompact(Page page, OffsetNumber offset);
extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
+extern void PageIndexClearWarmTuples(Page page, OffsetNumber *clearitemnos,
+ uint16 nclearitems);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
diff --git b/src/include/utils/rel.h a/src/include/utils/rel.h
index a617a7c..fbac7c0 100644
--- b/src/include/utils/rel.h
+++ a/src/include/utils/rel.h
@@ -138,9 +138,14 @@ typedef struct RelationData
/* data managed by RelationGetIndexAttrBitmap: */
Bitmapset *rd_indexattr; /* identifies columns used in indexes */
+ Bitmapset *rd_exprindexattr; /* indentified columns used in expression or
+ predicate indexes */
+ Bitmapset *rd_indxnotreadyattr; /* columns used by indexes not yet
+ ready */
Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */
Bitmapset *rd_pkattr; /* cols included in primary key */
Bitmapset *rd_idattr; /* included in replica identity index */
+ bool rd_supportswarm;/* True if the table can be WARM updated */
PublicationActions *rd_pubactions; /* publication actions */
diff --git b/src/include/utils/relcache.h a/src/include/utils/relcache.h
index da36b67..d18bd09 100644
--- b/src/include/utils/relcache.h
+++ a/src/include/utils/relcache.h
@@ -50,7 +50,9 @@ typedef enum IndexAttrBitmapKind
INDEX_ATTR_BITMAP_ALL,
INDEX_ATTR_BITMAP_KEY,
INDEX_ATTR_BITMAP_PRIMARY_KEY,
- INDEX_ATTR_BITMAP_IDENTITY_KEY
+ INDEX_ATTR_BITMAP_IDENTITY_KEY,
+ INDEX_ATTR_BITMAP_EXPR_PREDICATE,
+ INDEX_ATTR_BITMAP_NOTREADY
} IndexAttrBitmapKind;
extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation,
diff --git b/src/test/regress/expected/alter_generic.out a/src/test/regress/expected/alter_generic.out
index b01be59..37719c9 100644
--- b/src/test/regress/expected/alter_generic.out
+++ a/src/test/regress/expected/alter_generic.out
@@ -161,15 +161,15 @@ ALTER SERVER alt_fserv1 RENAME TO alt_fserv3; -- OK
SELECT fdwname FROM pg_foreign_data_wrapper WHERE fdwname like 'alt_fdw%';
fdwname
----------
- alt_fdw2
alt_fdw3
+ alt_fdw2
(2 rows)
SELECT srvname FROM pg_foreign_server WHERE srvname like 'alt_fserv%';
srvname
------------
- alt_fserv2
alt_fserv3
+ alt_fserv2
(2 rows)
--
diff --git b/src/test/regress/expected/rules.out a/src/test/regress/expected/rules.out
index bd13ae6..44c59ae 100644
--- b/src/test/regress/expected/rules.out
+++ a/src/test/regress/expected/rules.out
@@ -1732,6 +1732,7 @@ pg_stat_all_tables| SELECT c.oid AS relid,
pg_stat_get_tuples_updated(c.oid) AS n_tup_upd,
pg_stat_get_tuples_deleted(c.oid) AS n_tup_del,
pg_stat_get_tuples_hot_updated(c.oid) AS n_tup_hot_upd,
+ pg_stat_get_tuples_warm_updated(c.oid) AS n_tup_warm_upd,
pg_stat_get_live_tuples(c.oid) AS n_live_tup,
pg_stat_get_dead_tuples(c.oid) AS n_dead_tup,
pg_stat_get_mod_since_analyze(c.oid) AS n_mod_since_analyze,
@@ -1875,6 +1876,7 @@ pg_stat_sys_tables| SELECT pg_stat_all_tables.relid,
pg_stat_all_tables.n_tup_upd,
pg_stat_all_tables.n_tup_del,
pg_stat_all_tables.n_tup_hot_upd,
+ pg_stat_all_tables.n_tup_warm_upd,
pg_stat_all_tables.n_live_tup,
pg_stat_all_tables.n_dead_tup,
pg_stat_all_tables.n_mod_since_analyze,
@@ -1918,6 +1920,7 @@ pg_stat_user_tables| SELECT pg_stat_all_tables.relid,
pg_stat_all_tables.n_tup_upd,
pg_stat_all_tables.n_tup_del,
pg_stat_all_tables.n_tup_hot_upd,
+ pg_stat_all_tables.n_tup_warm_upd,
pg_stat_all_tables.n_live_tup,
pg_stat_all_tables.n_dead_tup,
pg_stat_all_tables.n_mod_since_analyze,
@@ -1955,7 +1958,8 @@ pg_stat_xact_all_tables| SELECT c.oid AS relid,
pg_stat_get_xact_tuples_inserted(c.oid) AS n_tup_ins,
pg_stat_get_xact_tuples_updated(c.oid) AS n_tup_upd,
pg_stat_get_xact_tuples_deleted(c.oid) AS n_tup_del,
- pg_stat_get_xact_tuples_hot_updated(c.oid) AS n_tup_hot_upd
+ pg_stat_get_xact_tuples_hot_updated(c.oid) AS n_tup_hot_upd,
+ pg_stat_get_xact_tuples_warm_updated(c.oid) AS n_tup_warm_upd
FROM ((pg_class c
LEFT JOIN pg_index i ON ((c.oid = i.indrelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
@@ -1971,7 +1975,8 @@ pg_stat_xact_sys_tables| SELECT pg_stat_xact_all_tables.relid,
pg_stat_xact_all_tables.n_tup_ins,
pg_stat_xact_all_tables.n_tup_upd,
pg_stat_xact_all_tables.n_tup_del,
- pg_stat_xact_all_tables.n_tup_hot_upd
+ pg_stat_xact_all_tables.n_tup_hot_upd,
+ pg_stat_xact_all_tables.n_tup_warm_upd
FROM pg_stat_xact_all_tables
WHERE ((pg_stat_xact_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_stat_xact_all_tables.schemaname ~ '^pg_toast'::text));
pg_stat_xact_user_functions| SELECT p.oid AS funcid,
@@ -1993,7 +1998,8 @@ pg_stat_xact_user_tables| SELECT pg_stat_xact_all_tables.relid,
pg_stat_xact_all_tables.n_tup_ins,
pg_stat_xact_all_tables.n_tup_upd,
pg_stat_xact_all_tables.n_tup_del,
- pg_stat_xact_all_tables.n_tup_hot_upd
+ pg_stat_xact_all_tables.n_tup_hot_upd,
+ pg_stat_xact_all_tables.n_tup_warm_upd
FROM pg_stat_xact_all_tables
WHERE ((pg_stat_xact_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_stat_xact_all_tables.schemaname !~ '^pg_toast'::text));
pg_statio_all_indexes| SELECT c.oid AS relid,
diff --git b/src/test/regress/expected/warm.out a/src/test/regress/expected/warm.out
new file mode 100644
index 0000000..6391891
--- /dev/null
+++ a/src/test/regress/expected/warm.out
@@ -0,0 +1,367 @@
+CREATE TABLE updtst_tab1 (a integer unique, b int, c text, d text);
+CREATE INDEX updtst_indx1 ON updtst_tab1 (b);
+INSERT INTO updtst_tab1
+ SELECT generate_series(1,10000), generate_series(70001, 80000), 'foo', 'bar';
+-- This should be a HOT update as non-index key is updated, but the
+-- page won't have any free space, so probably a non-HOT update
+UPDATE updtst_tab1 SET c = 'foo1' WHERE a = 1;
+-- Next update should be a HOT update as dead space is recycled
+UPDATE updtst_tab1 SET c = 'foo2' WHERE a = 1;
+-- And next too
+UPDATE updtst_tab1 SET c = 'foo3' WHERE a = 1;
+-- Now update one of the index key columns
+UPDATE updtst_tab1 SET b = b + 70000 WHERE a = 1;
+-- Ensure that the correct row is fetched
+SELECT * FROM updtst_tab1 WHERE a = 1;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+-- Even when seqscan is disabled and indexscan is forced
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab1
+ Recheck Cond: (b = 140001)
+ -> Bitmap Index Scan on updtst_indx1
+ Index Cond: (b = 140001)
+(4 rows)
+
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+ a | b | c | d
+---+--------+------+-----
+ 1 | 140001 | foo3 | bar
+(1 row)
+
+-- Check if index only scan works correctly
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab1
+ Recheck Cond: (b = 140001)
+ -> Bitmap Index Scan on updtst_indx1
+ Index Cond: (b = 140001)
+(4 rows)
+
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ b
+--------
+ 140001
+(1 row)
+
+-- Table must be vacuumed to force index-only scan
+VACUUM updtst_tab1;
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx1 on updtst_tab1
+ Index Cond: (b = 140001)
+(2 rows)
+
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+ b
+--------
+ 140001
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab1;
+------------------
+CREATE TABLE updtst_tab2 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx2 ON updtst_tab2 (b);
+INSERT INTO updtst_tab2
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+UPDATE updtst_tab2 SET b = b + 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo1' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab2 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo6' WHERE a = 1;
+SELECT count(*) FROM updtst_tab2 WHERE c = 'foo';
+ count
+-------
+ 99
+(1 row)
+
+SELECT * FROM updtst_tab2 WHERE c = 'foo6';
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab2
+ Recheck Cond: (b = 701)
+ -> Bitmap Index Scan on updtst_indx2
+ Index Cond: (b = 701)
+(4 rows)
+
+SELECT * FROM updtst_tab2 WHERE a = 1;
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+-----------------------------------------
+ Bitmap Heap Scan on updtst_tab2
+ Recheck Cond: (b = 701)
+ -> Bitmap Index Scan on updtst_indx2
+ Index Cond: (b = 701)
+(4 rows)
+
+SELECT * FROM updtst_tab2 WHERE b = 701;
+ a | b | c | d
+---+-----+------+-----
+ 1 | 701 | foo6 | bar
+(1 row)
+
+VACUUM updtst_tab2;
+EXPLAIN (costs off) SELECT b FROM updtst_tab2 WHERE b = 701;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx2 on updtst_tab2
+ Index Cond: (b = 701)
+(2 rows)
+
+SELECT b FROM updtst_tab2 WHERE b = 701;
+ b
+-----
+ 701
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab2;
+------------------
+CREATE TABLE updtst_tab3 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx3 ON updtst_tab3 (b);
+INSERT INTO updtst_tab3
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo1', b = b + 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo6' WHERE a = 1;
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo11', b = b + 750 WHERE b = 701;
+UPDATE updtst_tab3 SET c = 'foo12' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 1;
+COMMIT;
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+ count
+-------
+ 99
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo6';
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo12';
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE a = 1;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+ a | b | c | d
+---+------+-------+-----
+ 1 | 1421 | foo12 | bar
+(1 row)
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 701;
+ QUERY PLAN
+-------------------------
+ Seq Scan on updtst_tab3
+ Filter: (b = 701)
+(2 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 701;
+ b
+---
+(0 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 1421;
+ b
+------
+ 1421
+(1 row)
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo23' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo24' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo25' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo26' WHERE a = 2;
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+SET enable_seqscan = false;
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 750 WHERE b = 702;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 2;
+COMMIT;
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+ count
+-------
+ 98
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo26';
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE c = 'foo22';
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE b = 702;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+SELECT * FROM updtst_tab3 WHERE a = 2;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+-- Try fetching both old and new value using updtst_indx3
+SELECT * FROM updtst_tab3 WHERE b = 702;
+ a | b | c | d
+---+---+---+---
+(0 rows)
+
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+ a | b | c | d
+---+------+-------+-----
+ 2 | 1422 | foo22 | bar
+(1 row)
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 702;
+ QUERY PLAN
+---------------------------------------------------
+ Index Only Scan using updtst_indx3 on updtst_tab3
+ Index Cond: (b = 702)
+(2 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 702;
+ b
+---
+(0 rows)
+
+SELECT b FROM updtst_tab3 WHERE b = 1422;
+ b
+------
+ 1422
+(1 row)
+
+SET enable_seqscan = true;
+DROP TABLE updtst_tab3;
+------------------
+CREATE TABLE test_warm (a text unique, b text);
+CREATE INDEX test_warmindx ON test_warm (lower(a));
+INSERT INTO test_warm values ('test', 'foo');
+UPDATE test_warm SET a = 'TEST';
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+explain select * from test_warm where lower(a) = 'test';
+ QUERY PLAN
+----------------------------------------------------------------------------
+ Bitmap Heap Scan on test_warm (cost=4.18..12.65 rows=4 width=64)
+ Recheck Cond: (lower(a) = 'test'::text)
+ -> Bitmap Index Scan on test_warmindx (cost=0.00..4.18 rows=4 width=0)
+ Index Cond: (lower(a) = 'test'::text)
+(4 rows)
+
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+select *, ctid from test_warm where a = 'test';
+ a | b | ctid
+---+---+------
+(0 rows)
+
+select *, ctid from test_warm where a = 'TEST';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+set enable_bitmapscan TO false;
+explain select * from test_warm where lower(a) = 'test';
+ QUERY PLAN
+---------------------------------------------------------------------------------
+ Index Scan using test_warmindx on test_warm (cost=0.15..20.22 rows=4 width=64)
+ Index Cond: (lower(a) = 'test'::text)
+(2 rows)
+
+select *, ctid from test_warm where lower(a) = 'test';
+ a | b | ctid
+------+-----+-------
+ TEST | foo | (0,2)
+(1 row)
+
+DROP TABLE test_warm;
diff --git b/src/test/regress/parallel_schedule a/src/test/regress/parallel_schedule
index ea7b5b4..7cc0d21 100644
--- b/src/test/regress/parallel_schedule
+++ a/src/test/regress/parallel_schedule
@@ -42,6 +42,8 @@ test: create_type
test: create_table
test: create_function_2
+test: warm
+
# ----------
# Load huge amounts of data
# We should split the data files into single files and then
diff --git b/src/test/regress/sql/warm.sql a/src/test/regress/sql/warm.sql
new file mode 100644
index 0000000..3a078dd
--- /dev/null
+++ a/src/test/regress/sql/warm.sql
@@ -0,0 +1,170 @@
+
+CREATE TABLE updtst_tab1 (a integer unique, b int, c text, d text);
+CREATE INDEX updtst_indx1 ON updtst_tab1 (b);
+INSERT INTO updtst_tab1
+ SELECT generate_series(1,10000), generate_series(70001, 80000), 'foo', 'bar';
+
+-- This should be a HOT update as non-index key is updated, but the
+-- page won't have any free space, so probably a non-HOT update
+UPDATE updtst_tab1 SET c = 'foo1' WHERE a = 1;
+
+-- Next update should be a HOT update as dead space is recycled
+UPDATE updtst_tab1 SET c = 'foo2' WHERE a = 1;
+
+-- And next too
+UPDATE updtst_tab1 SET c = 'foo3' WHERE a = 1;
+
+-- Now update one of the index key columns
+UPDATE updtst_tab1 SET b = b + 70000 WHERE a = 1;
+
+-- Ensure that the correct row is fetched
+SELECT * FROM updtst_tab1 WHERE a = 1;
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Even when seqscan is disabled and indexscan is forced
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT * FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Check if index only scan works correctly
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+-- Table must be vacuumed to force index-only scan
+VACUUM updtst_tab1;
+EXPLAIN (costs off) SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+SELECT b FROM updtst_tab1 WHERE b = 70001 + 70000;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab1;
+
+------------------
+
+CREATE TABLE updtst_tab2 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx2 ON updtst_tab2 (b);
+INSERT INTO updtst_tab2
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+
+UPDATE updtst_tab2 SET b = b + 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo1' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab2 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab2 SET c = 'foo6' WHERE a = 1;
+
+SELECT count(*) FROM updtst_tab2 WHERE c = 'foo';
+SELECT * FROM updtst_tab2 WHERE c = 'foo6';
+
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+SELECT * FROM updtst_tab2 WHERE a = 1;
+
+SET enable_seqscan = false;
+EXPLAIN (costs off) SELECT * FROM updtst_tab2 WHERE b = 701;
+SELECT * FROM updtst_tab2 WHERE b = 701;
+
+VACUUM updtst_tab2;
+EXPLAIN (costs off) SELECT b FROM updtst_tab2 WHERE b = 701;
+SELECT b FROM updtst_tab2 WHERE b = 701;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab2;
+------------------
+
+CREATE TABLE updtst_tab3 (a integer unique, b int, c text, d text) WITH (fillfactor = 80);
+CREATE INDEX updtst_indx3 ON updtst_tab3 (b);
+INSERT INTO updtst_tab3
+ SELECT generate_series(1,100), generate_series(701, 800), 'foo', 'bar';
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo1', b = b + 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo2' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo3' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo4' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo5' WHERE a = 1;
+UPDATE updtst_tab3 SET c = 'foo6' WHERE a = 1;
+
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo11', b = b + 750 WHERE b = 701;
+UPDATE updtst_tab3 SET c = 'foo12' WHERE a = 1;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 1;
+COMMIT;
+
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+SELECT * FROM updtst_tab3 WHERE c = 'foo6';
+SELECT * FROM updtst_tab3 WHERE c = 'foo12';
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+SELECT * FROM updtst_tab3 WHERE a = 1;
+
+SELECT * FROM updtst_tab3 WHERE b = 701;
+SELECT * FROM updtst_tab3 WHERE b = 1421;
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 701;
+SELECT b FROM updtst_tab3 WHERE b = 701;
+SELECT b FROM updtst_tab3 WHERE b = 1421;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo23' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 700 WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo24' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo25' WHERE a = 2;
+UPDATE updtst_tab3 SET c = 'foo26' WHERE a = 2;
+
+-- Abort the transaction and ensure the original tuple is visible correctly
+ROLLBACK;
+
+SET enable_seqscan = false;
+
+BEGIN;
+UPDATE updtst_tab3 SET c = 'foo21', b = b + 750 WHERE b = 702;
+UPDATE updtst_tab3 SET c = 'foo22' WHERE a = 2;
+UPDATE updtst_tab3 SET b = b - 30 WHERE a = 2;
+COMMIT;
+
+SELECT count(*) FROM updtst_tab3 WHERE c = 'foo';
+SELECT * FROM updtst_tab3 WHERE c = 'foo26';
+SELECT * FROM updtst_tab3 WHERE c = 'foo22';
+
+SELECT * FROM updtst_tab3 WHERE b = 702;
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+SELECT * FROM updtst_tab3 WHERE a = 2;
+
+-- Try fetching both old and new value using updtst_indx3
+SELECT * FROM updtst_tab3 WHERE b = 702;
+SELECT * FROM updtst_tab3 WHERE b = 1422;
+
+VACUUM updtst_tab3;
+EXPLAIN (costs off) SELECT b FROM updtst_tab3 WHERE b = 702;
+SELECT b FROM updtst_tab3 WHERE b = 702;
+SELECT b FROM updtst_tab3 WHERE b = 1422;
+
+SET enable_seqscan = true;
+
+DROP TABLE updtst_tab3;
+------------------
+
+CREATE TABLE test_warm (a text unique, b text);
+CREATE INDEX test_warmindx ON test_warm (lower(a));
+INSERT INTO test_warm values ('test', 'foo');
+UPDATE test_warm SET a = 'TEST';
+select *, ctid from test_warm where lower(a) = 'test';
+explain select * from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where a = 'test';
+select *, ctid from test_warm where a = 'TEST';
+set enable_bitmapscan TO false;
+explain select * from test_warm where lower(a) = 'test';
+select *, ctid from test_warm where lower(a) = 'test';
+DROP TABLE test_warm;
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 10:24 Pavan Deolasee <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 10:24 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Thanks Amit. v19 addresses some of the comments below.
On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila <[email protected]>
wrote:
> On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila <[email protected]>
> wrote:
> > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee
> > <[email protected]> wrote:
> >>
> >>>
> >>
> >> Please find attached rebased patches.
> >>
> >
> > Few comments on 0005_warm_updates_v18.patch:
> >
>
> Few more comments on 0005_warm_updates_v18.patch:
> 1.
> @@ -234,6 +241,25 @@ index_beginscan(Relation heapRelation,
> scan->heapRelation = heapRelation;
> scan->xs_snapshot = snapshot;
>
> + /*
> + * If the index supports recheck,
> make sure that index tuple is saved
> + * during index scans. Also build and cache IndexInfo which is used by
> + * amrecheck routine.
> + *
> + * XXX Ideally, we should look at
> all indexes on the table and check if
> + * WARM is at all supported on the base table. If WARM is not supported
> + * then we don't need to do any recheck.
> RelationGetIndexAttrBitmap() does
> + * do that and sets rd_supportswarm after looking at all indexes. But we
> + * don't know if the function was called earlier in the
> session when we're
> + * here. We can't call it now because there exists a risk of causing
> + * deadlock.
> + */
> + if (indexRelation->rd_amroutine->amrecheck)
> + {
> +scan->xs_want_itup = true;
> + scan->indexInfo = BuildIndexInfo(indexRelation);
> + }
> +
>
> Don't we need to do this rechecking during parallel scans? Also what
> about bitmap heap scans?
>
>
Yes, we need to handle parallel scans. Bitmap scans are not a problem
because it can never return the same TID twice. I fixed this though by
moving this inside index_beginscan_internal.
> 2.
> +++ b/src/backend/access/nbtree/nbtinsert.c
> -
> typedef struct
>
> Above change is not require.
>
>
Sure. Fixed.
> 3.
> +_bt_clear_items(Page page, OffsetNumber *clearitemnos, uint16 nclearitems)
> +void _hash_clear_items(Page page, OffsetNumber *clearitemnos,
> + uint16 nclearitems)
>
> Both the above functions look exactly same, isn't it better to have a
> single function like page_clear_items? If you want separation for
> different index types, then we can have one common function which can
> be called from different index types.
>
>
Yes, makes sense. Moved that to bufpage.c. The reason why I originally had
index-specific versions because I started by putting WARM flag in
IndexTuple header. But since hash index does not have the bit free, moved
everything to TID bit-flag. I still left index-specific wrappers, but they
just call PageIndexClearWarmTuples.
> 4.
> - if (callback(htup, callback_state))
> + flags = ItemPointerGetFlags(&itup->t_tid);
> + is_warm = ((flags & BTREE_INDEX_WARM_POINTER) != 0);
> +
> + if (is_warm)
> + stats->num_warm_pointers++;
> + else
> + stats->num_clear_pointers++;
> +
> + result = callback(htup, is_warm, callback_state);
> + if (result == IBDCR_DELETE)
> + {
> + if (is_warm)
> + stats->warm_pointers_removed++;
> + else
> + stats->clear_pointers_removed++;
>
> The patch looks to be inconsistent in collecting stats for btree and
> hash. I don't see above stats are getting updated in hash index code.
>
>
Fixed. The hashbucketcleanup signature is just getting a bit too long. May
be we should move some of these counters in a structure and pass that
around. Not done here though.
> 5.
> +btrecheck(Relation indexRel, IndexInfo *indexInfo, IndexTuple indexTuple,
> + Relation heapRel, HeapTuple heapTuple)
> {
> ..
> + if (!datumIsEqual(values[i - 1], indxvalue, att->attbyval,
> + att->attlen))
> ..
> }
>
> Will this work if the index is using non-default collation?
>
>
Not sure I understand that. Can you please elaborate?
> 6.
> +++ b/src/backend/access/nbtree/nbtxlog.c
> @@ -390,83 +390,9 @@ btree_xlog_vacuum(XLogReaderState *record)
> -#ifdef UNUSED
> xl_btree_vacuum *xlrec = (xl_btree_vacuum *) XLogRecGetData(record);
>
> /*
> - * This section of code is thought to be no longer needed, after analysis
> - * of the calling paths. It is retained to allow the code to be reinstated
> - * if a flaw is revealed in that thinking.
> - *
> ..
>
> Why does this patch need to remove the above code under #ifdef UNUSED
>
>
Yeah, it isn't strictly necessary. But that dead code was coming in the way
and hence I decided to strip it out. I can put it back if it's an issue or
remove that as a separate commit first.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 10:38 Pavan Deolasee <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 10:38 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mithun Cy <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila <[email protected]>
wrote:
>
>
> That sounds like you are dodging the actual problem. I mean you can
> put that same PageIsFull() check in master code as well and then you
> will most probably again see the same regression.
Well I don't see it that way. There was a specific concern about a specific
workload that WARM might regress. I think this change addresses that. Sure
if you pick that one piece, put it in master first and then compare against
rest of the WARM code, you will see a regression. But I thought what we
were worried is WARM causing regression to some existing user, who might
see her workload running 10% slower, which this change mitigates.
> Also, I think if we
> test at fillfactor 80 or 75 (which is not unrealistic considering an
> update-intensive workload), then we might again see regression.
Yeah, we might, but it will be lesser than before, may be 2% instead of
10%. And by doing this we are further narrowing an already narrow test
case. I think we need to see things in totality and weigh in costs-benefits
trade offs. There are numbers for very common workloads, where WARM may
provide 20, 30 or even more than 100% improvements.
Andres and Alvaro already have other ideas to address this problem even
further. And as I said, we can pass-in index specific information and make
that routine bail-out even earlier. We need to accept that WARM will need
to do more attr checks than master, especially when there are more than 1
indexes on the table, and sometimes those checks will go waste. I am ok if
we want to provide table-specific knob to disable WARM, but not sure if
others would like that idea.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 11:34 Pavan Deolasee <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 11:34 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mithun Cy <[email protected]>; Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 4:08 PM, Pavan Deolasee <[email protected]>
wrote:
>
>
> On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila <[email protected]>
> wrote:
>
>>
>>
>> That sounds like you are dodging the actual problem. I mean you can
>> put that same PageIsFull() check in master code as well and then you
>> will most probably again see the same regression.
>
>
> Well I don't see it that way. There was a specific concern about a
> specific workload that WARM might regress. I think this change addresses
> that. Sure if you pick that one piece, put it in master first and then
> compare against rest of the WARM code, you will see a regression.
>
BTW the PageIsFull() check may not help as much in master as it does with
WARM. In master we anyways bail out early after couple of column checks. In
master it may help to reduce the 10% drop that we see while updating last
index column, but if we compare master and WARM with the patch applied,
regression should be quite nominal.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 14:23 Amit Kapila <[email protected]>
parent: Pavan Deolasee <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Amit Kapila @ 2017-03-23 14:23 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee
<[email protected]> wrote:
> On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila <[email protected]>
> wrote:
>>
>
>> 3.
>> + /*
>> + * HASH indexes compute a hash value of the key and store that in the
>> + * index. So
>> we must first obtain the hash of the value obtained from the
>> + * heap and then do a comparison
>> +
>> */
>> + _hash_convert_tuple(indexRel, values, isnull, values2, isnull2);
>>
>> I think here, you need to handle the case where heap has a NULL value
>> as the hash index doesn't contain NULL values, otherwise, the code in
>> below function can return true which is not right.
>>
>
> I think we can simply conclude hashrecheck has failed the equality if the
> heap has NULL value because such a tuple should not have been reached via
> hash index unless a non-NULL hash key was later updated to a NULL key,
> right?
>
Right.
>>
>>
>> 6.
>> + *stats = index_bulk_delete(&ivinfo, *stats,
>> +lazy_indexvac_phase1, (void *) vacrelstats);
>> + ereport(elevel,
>> +(errmsg("scanned index \"%s\" to remove %d row version, found "
>> +"%0.f warm pointers, %0.f clear pointers, removed "
>> +"%0.f warm pointers, removed %0.f clear pointers",
>> +RelationGetRelationName(indrel),
>> + vacrelstats->num_dead_tuples,
>> + (*stats)->num_warm_pointers,
>> +(*stats)->num_clear_pointers,
>> +(*stats)->warm_pointers_removed,
>> + (*stats)->clear_pointers_removed)));
>> +
>> + (*stats)->num_warm_pointers = 0;
>> + (*stats)->num_clear_pointers = 0;
>> + (*stats)->warm_pointers_removed = 0;
>> + (*stats)->clear_pointers_removed = 0;
>> + (*stats)->pointers_cleared = 0;
>> +
>> + *stats =index_bulk_delete(&ivinfo, *stats,
>> + lazy_indexvac_phase2, (void *)vacrelstats);
>>
>> To convert WARM chains, we need to do two index passes for all the
>> indexes. I think it can substantially increase the random I/O. I
>> think this can help us in doing more WARM updates, but I don't see how
>> the downside of that (increased random I/O) will be acceptable for all
>> kind of cases.
>>
>
> Yes, this is a very fair point. The way I proposed to address this upthread
> is by introducing a set of threshold/scale GUCs specific to WARM. So users
> can control when to invoke WARM cleanup. Only if the WARM cleanup is
> required, we do 2 index scans. Otherwise vacuum will work the way it works
> today without any additional overhead.
>
I am not sure on what basis user can set such parameters, it will be
quite difficult to tune those parameters. I think the point is
whatever threshold we keep, once that is crossed, it will perform two
scans for all the indexes. IIUC, this conversion of WARM chains is
required so that future updates can be WARM or is there any other
reason? I see this as a big penalty for future updates.
> We already have some intelligence to skip the second index scan if we did
> not find any WARM candidate chains during the first heap scan. This should
> take care of majority of the users who never update their indexed columns.
> For others, we need either a knob or some built-in way to deduce whether to
> do WARM cleanup or not.
>
> Does that seem worthwhile?
>
Is there any consensus on your proposal, because I feel this needs
somewhat broader discussion, me and you can't take a call on this
point. I request others also to share their opinion on this point.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 18:14 Mithun Cy <[email protected]>
parent: Pavan Deolasee <[email protected]>
2 siblings, 1 reply; 104+ messages in thread
From: Mithun Cy @ 2017-03-23 18:14 UTC (permalink / raw)
To: Pavan Deolasee <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
Hi Pavan,
On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee
<[email protected]> wrote:
> Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB
> RAM, attached SSD) and results are shown below. But I think it is important
> to get independent validation from your side too, just to ensure I am not
> making any mistake in measurement. I've attached naively put together
> scripts which I used to run the benchmark. If you find them useful, please
> adjust the paths and run on your machine.
I did a similar test appears. Your v19 looks fine to me, it does not
cause any regression, On the other hand, I also ran tests reducing
table fillfactor to 80 there I can see a small regression 2-3% in
average when updating col2 and on updating col9 again I do not see any
regression.
--
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/vnd.oasis.opendocument.spreadsheet] WARM_test_02.ods (14.9K, ../../CAD__OujY23cbohBb7=eh3PYWda-p4FK3Ne-_Jsj9xXNTwJBnMQ@mail.gmail.com/2-WARM_test_02.ods)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 18:35 Pavan Deolasee <[email protected]>
parent: Mithun Cy <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 18:35 UTC (permalink / raw)
To: Mithun Cy <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 11:44 PM, Mithun Cy <[email protected]>
wrote:
> Hi Pavan,
> On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee
> <[email protected]> wrote:
> > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB
> > RAM, attached SSD) and results are shown below. But I think it is
> important
> > to get independent validation from your side too, just to ensure I am not
> > making any mistake in measurement. I've attached naively put together
> > scripts which I used to run the benchmark. If you find them useful,
> please
> > adjust the paths and run on your machine.
>
> I did a similar test appears. Your v19 looks fine to me, it does not
> cause any regression, On the other hand, I also ran tests reducing
> table fillfactor to 80 there I can see a small regression 2-3% in
> average when updating col2 and on updating col9 again I do not see any
> regression.
>
>
Thanks Mithun for repeating the tests and confirming that the v19 patch
looks ok.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 18:47 Pavan Deolasee <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 18:47 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Wed, Mar 22, 2017 at 12:30 AM, Bruce Momjian <[email protected]> wrote:
>
>
> Well, it is really a question of how often you want to do a second WARM
> update (not possible) vs. the frequency of lazy vacuum. I assumed that
> would be a 100X or 10kX difference, but I am not sure myself either. My
> initial guess was that only allowing a single WARM update between lazy
> vacuums would show no improvementin in real-world workloads, but maybe I
> am wrong.
>
>
It's quite hard to say that until we see many more benchmarks. As author of
the patch, I might have got repetitive with my benchmarks. But I've seen
over 50% improvement in TPS even without chain conversion (6 indexes on a
12 column table test).
With chain conversion, in my latest tests, I saw over 100% improvement. The
benchmark probably received between 6-8 autovac cycles in an 8hr test. This
was with a large table which doesn't fit in memory or barely fit in memory.
Graphs attached again just in case you missed (x-axis test duration in
seconds, y-axis moving average of TPS)
May be we should run another set with just 2 or 3 indexes on a 12 column
table and see how much that helps, if at all. Or may be do a mix of HOT and
WARM updates. Or even just do HOT updates on small and large tables and
look for any regression. Will try to schedule some of those tests.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/pdf] Moderate_AV_4Indexes_100FF_SF800_Duration28800s.pdf (248.7K, ../../CABOikdPpm2AeUsS9=BEs=cUoRV7LS6cMQ+WHBjUzfvO+pctBtw@mail.gmail.com/3-Moderate_AV_4Indexes_100FF_SF800_Duration28800s.pdf)
download
[application/pdf] Moderate_AV_4Indexes_100FF_SF1200_Duration28800s_Run2.pdf (225.5K, ../../CABOikdPpm2AeUsS9=BEs=cUoRV7LS6cMQ+WHBjUzfvO+pctBtw@mail.gmail.com/4-Moderate_AV_4Indexes_100FF_SF1200_Duration28800s_Run2.pdf)
download
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: Patch: Write Amplification Reduction Method (WARM)
@ 2017-03-23 18:55 Pavan Deolasee <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Pavan Deolasee @ 2017-03-23 18:55 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Jaime Casanova <[email protected]>; Haribabu Kommi <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila <[email protected]>
wrote:
> On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee
>
> >
> > Yes, this is a very fair point. The way I proposed to address this
> upthread
> > is by introducing a set of threshold/scale GUCs specific to WARM. So
> users
> > can control when to invoke WARM cleanup. Only if the WARM cleanup is
> > required, we do 2 index scans. Otherwise vacuum will work the way it
> works
> > today without any additional overhead.
> >
>
> I am not sure on what basis user can set such parameters, it will be
> quite difficult to tune those parameters. I think the point is
> whatever threshold we keep, once that is crossed, it will perform two
> scans for all the indexes.
Well, that applies to even vacuum parameters, no? The general sense I've
got here is that we're ok to push some work in background if it helps the
real-time queries, and I kinda agree with that. If WARM improves things in
a significant manner even with these additional maintenance work, it's
still worth doing.
Having said that, I see many ways we can improve on this later. Like we can
track somewhere else information about tuples which may have received WARM
updates (I think it will need to be a per-index bitmap or so) and use that
to do WARM chain conversion in a single index pass. But this is clearly not
PG 10 material.
> IIUC, this conversion of WARM chains is
> required so that future updates can be WARM or is there any other
> reason? I see this as a big penalty for future updates.
>
It's also necessary for index-only-scans. But I don't see this as a big
penalty for future updates, because if there are indeed significant WARM
updates then not preparing for future updates will result in
write-amplification, something we are trying to solve here and something
which seems to be showing good gains.
Thanks,
Pavan
--
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v4 04/12] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index a9bc397165..d58cd05f46 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -323,7 +323,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option reduces the duration of the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index f6c3d9538b..4388d1329c 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -173,8 +173,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index fd1dc140ab..93a3eed813 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing
+ duration but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--DiL7RhKs8rK9YGuF
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0005-docs-Add-wait-events-for-recovery-conflicts.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v3 05/20] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index a9bc397165..d58cd05f46 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -323,7 +323,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option reduces the duration of the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index f6c3d9538b..4388d1329c 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -173,8 +173,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index fd1dc140ab..93a3eed813 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing
+ duration but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--doUn1Hmx68n+7ij2
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0006-docs-Add-wait-events-for-recovery-conflicts.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v2 06/12] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index a9bc397165..d58cd05f46 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -323,7 +323,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option reduces the duration of the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index f6c3d9538b..4388d1329c 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -173,8 +173,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index fd1dc140ab..93a3eed813 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing
+ duration but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--xYeFQzU4VZLrHqxU
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0007-docs-Add-wait-events-for-recovery-conflicts.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v1 12/19] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index a9bc397165..d58cd05f46 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -323,7 +323,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option reduces the duration of the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index f6c3d9538b..4388d1329c 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -173,8 +173,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index fd1dc140ab..93a3eed813 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option reduces the processing
+ duration but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--4ybNbZnZ8tziJ7D6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v1-0013-docs-Add-wait-events-for-recovery-conflicts.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v6 03/10] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..77029defc5 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduces the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--C+ts3FVlLX8+P6JN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-doc-Allow-users-to-limit-storage-reserved-by-repl.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* [PATCH v9 10/15] Fix docs: "time of the"
@ 2020-04-06 03:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Justin Pryzby @ 2020-04-06 03:35 UTC (permalink / raw)
commit 9e257a181cc1dc5e19eb5d770ce09cc98f470f5f
Author: Andrew Dunstan <[email protected]>
Date: Sun Mar 24 11:27:20 2013 -0400
Add parallel pg_dump option.
commit 5ab892c391c6bc54a00e7a8de5cab077cabace6a
Author: Michael Paquier <[email protected]>
Date: Sat Jul 27 22:21:18 2019 +0900
Add support for --jobs in reindexdb
commit a17923204736d8842eade3517d6a8ee81290fca4
Author: Alvaro Herrera <[email protected]>
Date: Fri Jan 23 15:02:45 2015 -0300
vacuumdb: enable parallel mode
---
doc/src/sgml/ref/pg_dump.sgml | 2 +-
doc/src/sgml/ref/reindexdb.sgml | 4 ++--
doc/src/sgml/ref/vacuumdb.sgml | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0ec99165c5..e09ed0a4c3 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -322,7 +322,7 @@ PostgreSQL documentation
<listitem>
<para>
Run the dump in parallel by dumping <replaceable class="parameter">njobs</replaceable>
- tables simultaneously. This option reduces the time of the dump but it also
+ tables simultaneously. This option may reduce the time needed to perform the dump but it also
increases the load on the database server. You can only use this option with the
directory output format because this is the only output format where multiple processes
can write their data at the same time.
diff --git a/doc/src/sgml/ref/reindexdb.sgml b/doc/src/sgml/ref/reindexdb.sgml
index 026fd018d9..4f821c2095 100644
--- a/doc/src/sgml/ref/reindexdb.sgml
+++ b/doc/src/sgml/ref/reindexdb.sgml
@@ -174,8 +174,8 @@ PostgreSQL documentation
<para>
Execute the reindex commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>reindexdb</application> will open
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 95d6894cb0..7b13950552 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -155,8 +155,8 @@ PostgreSQL documentation
<para>
Execute the vacuum or analyze commands in parallel by running
<replaceable class="parameter">njobs</replaceable>
- commands simultaneously. This option reduces the time of the
- processing but it also increases the load on the database server.
+ commands simultaneously. This option may reduce the processing time
+ but it also increases the load on the database server.
</para>
<para>
<application>vacuumdb</application> will open
--
2.17.0
--L6iaP+gRLNZHKoI4
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0011-s-periodical-periodic.patch"
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: doing also VM cache snapshot and restore with pg_prewarm, having more information of the VM inside PostgreSQL
@ 2024-01-04 22:41 Jim Nasby <[email protected]>
0 siblings, 1 reply; 104+ messages in thread
From: Jim Nasby @ 2024-01-04 22:41 UTC (permalink / raw)
To: Cedric Villemain <[email protected]>; pgsql-hackers
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 1/3/24 5:57 PM, Cedric Villemain
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<p>for 15 years pgfincore has been sitting quietly and being used
in large setups to help in HA and resources management.<br>
It can perfectly stay as is, to be honest I was expecting to one
day include a windows support and propose that to PostgreSQL, it
appears getting support on linux and BSD is more than enough
today.</p>
<p>So I wonder if there are interest for having virtual memory
snapshot and restore operations with, for example,
pg_prewarm/autowarm ?<br>
</p>
</blockquote>
<p>IMHO, to improve the user experience here we'd need something
that combined the abilities of all these extensions into a
cohesive interface that allowed users to simply say "please get
this data into cache". Simply moving pgfincore into core Postgres
wouldn't satisfy that need.</p>
<p>So I think the real question is whether the community feels
spport for better cache (buffercache + filesystem) management is a
worthwhile feature to add to Postgres.</p>
<p>Micromanaging cache contents for periodic jobs seems almost like
a mis-feature. While it's a useful tool to have in the toolbox,
it's also a non-trivial layer of complexity. IMHO not something
we'd want to add. Though, there might be smaller items that would
make creating tools to do that easier, such as some ability to see
what blocks a backend is accessing (perhaps via a hook).</p>
<p>On the surface, improving RTO via cache warming sounds
interesting ... but I'm not sure how useful it would be in
reality. Users that care about RTO would almost always have some
form of hot-standby, and generally those will already have a lot
of data in cache. While they won't have read-only data in cache, I
have to wonder if the answer to that problem is allowing writers
to tell a replica what blocks are being read, so the replica can
keep them in cache. Also, most (all?) operations that require a
restart could be handled via a failover, so I'm not sure how much
cache management moves the needle there.<br>
</p>
<blockquote type="cite"
cite="mid:[email protected]">
<p> </p>
<p>Some usecases covered: snapshot/restore cache around cronjobs,
around dumps, switchover, failover, on stop/start of postgres
(think kernel upgrade with a cold restart), ...<br>
<br>
</p>
<p>pgfincore also provides some nice information with mincore (on
FreeBSD mincore is more interesting) or cachestat, again it can
remain as an out of tree extension but I will be happy to add to
commitfest if there are interest from the community.<br>
An example of cachestat output:</p>
<p><span style="font-family:monospace"><span
style="color:#000000;background-color:#ffffff;">postgres=#
select *from vm_relation_cachestat('foo',range:=1024*32); </span><br>
block_start | block_count | nr_pages | nr_cache | nr_dirty |
nr_writeback | nr_evicted | nr_recently_evicted <br>
-------------+-------------+----------+----------+----------+--------------+------------+---------------------
<br>
0 | 32768 | 65536 | 62294 | 0 |
0 | 3242 | 3242 <br>
32768 | 32768 | 65536 | 39279 | 0 |
0 | 26257 | 26257 <br>
65536 | 32768 | 65536 | 22516 | 0 |
0 | 43020 | 43020 <br>
98304 | 32768 | 65536 | 24944 | 0 |
0 | 40592 | 40592 <br>
131072 | 1672 | 3344 | 487 | 0 |
0 | 2857 | 2857<br>
<br>
</span></p>
<p><span style="font-family:monospace"><br>
</span></p>
<p><span style="font-family:monospace">Comments?<br>
</span></p>
<pre class="moz-signature" cols="72">---
Cédric Villemain +33 (0)6 20 30 22 52
<a class="moz-txt-link-freetext" href="https://Data-Bene.io";
moz-do-not-send="true">https://Data-Bene.io</a;
PostgreSQL Expertise, Support, Training, R&D</pre>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
Jim Nasby, Data Architect, Austin TX</pre>
</body>
</html>
^ permalink raw reply [nested|flat] 104+ messages in thread
* Re: doing also VM cache snapshot and restore with pg_prewarm, having more information of the VM inside PostgreSQL
@ 2024-01-05 09:32 Cédric Villemain <[email protected]>
parent: Jim Nasby <[email protected]>
0 siblings, 0 replies; 104+ messages in thread
From: Cédric Villemain @ 2024-01-05 09:32 UTC (permalink / raw)
To: Jim Nasby <[email protected]>; pgsql-hackers
Le 04/01/2024 à 23:41, Jim Nasby a écrit :
> On 1/3/24 5:57 PM, Cedric Villemain wrote:
>>
>> for 15 years pgfincore has been sitting quietly and being used in
>> large setups to help in HA and resources management.
>> It can perfectly stay as is, to be honest I was expecting to one day
>> include a windows support and propose that to PostgreSQL, it appears
>> getting support on linux and BSD is more than enough today.
>>
>> So I wonder if there are interest for having virtual memory snapshot
>> and restore operations with, for example, pg_prewarm/autowarm ?
>>
> IMHO, to improve the user experience here we'd need something that
> combined the abilities of all these extensions into a cohesive interface
> that allowed users to simply say "please get this data into cache".
> Simply moving pgfincore into core Postgres wouldn't satisfy that need.
This is exactly why I proposed those additions to pg_prewarm and autowarm.
> So I think the real question is whether the community feels spport for
> better cache (buffercache + filesystem) management is a worthwhile
> feature to add to Postgres.
Agreed, to add in an extension more probably and only the "filesystem"
part as the buffercache is done already.
> Micromanaging cache contents for periodic jobs seems almost like a
> mis-feature. While it's a useful tool to have in the toolbox, it's also
> a non-trivial layer of complexity. IMHO not something we'd want to add.
> Though, there might be smaller items that would make creating tools to
> do that easier, such as some ability to see what blocks a backend is
> accessing (perhaps via a hook).
From my point of view it's not so complex, but this is subjective and I
won't argue in this area.
I confirm that having someway to get feedback on current position/block
activity (and distance/context: heap scan, index build, analyze, ...)
would be super useful to allow external management. Maybe the "progress"
facilities can be used for that. Maybe worth looking at that for another
proposal than the current one.
To be clear I am not proposing that PostgreSQL handles those tasks
transparently or itself, but offering options to the users via
extensions like we do with pg_prewarm and pg_buffercache.
It's just the same for virtual memory/filesystem.
---
Cédric Villemain +33 (0)6 20 30 22 52
https://Data-Bene.io
PostgreSQL Expertise, Support, Training, R&D
^ permalink raw reply [nested|flat] 104+ messages in thread
end of thread, other threads:[~2024-01-05 09:32 UTC | newest]
Thread overview: 104+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-03-08 17:00 Re: Patch: Write Amplification Reduction Method (WARM) Alvaro Herrera <[email protected]>
2017-03-08 17:08 ` Robert Haas <[email protected]>
2017-03-08 17:14 ` Alvaro Herrera <[email protected]>
2017-03-08 18:18 ` Robert Haas <[email protected]>
2017-03-08 19:30 ` Alvaro Herrera <[email protected]>
2017-03-09 03:13 ` Robert Haas <[email protected]>
2017-03-10 18:07 ` Alvaro Herrera <[email protected]>
2017-03-21 10:51 ` Amit Kapila <[email protected]>
2017-03-21 10:56 ` Amit Kapila <[email protected]>
2017-03-21 12:04 ` Robert Haas <[email protected]>
2017-03-21 12:41 ` Pavan Deolasee <[email protected]>
2017-03-21 13:25 ` Robert Haas <[email protected]>
2017-03-21 14:01 ` Amit Kapila <[email protected]>
2017-03-21 14:12 ` Robert Haas <[email protected]>
2017-03-21 14:21 ` Alvaro Herrera <[email protected]>
2017-03-21 14:40 ` Robert Haas <[email protected]>
2017-03-21 22:21 ` Mithun Cy <[email protected]>
2017-03-22 03:13 ` Pavan Deolasee <[email protected]>
2017-03-22 07:50 ` Pavan Deolasee <[email protected]>
2017-03-22 14:20 ` Mithun Cy <[email protected]>
2017-03-22 10:14 ` Pavan Deolasee <[email protected]>
2017-03-22 11:23 ` Mithun Cy <[email protected]>
2017-03-22 18:49 ` Pavan Deolasee <[email protected]>
2017-03-22 19:17 ` Mithun Cy <[email protected]>
2017-03-23 09:32 ` Amit Kapila <[email protected]>
2017-03-23 10:38 ` Pavan Deolasee <[email protected]>
2017-03-23 11:34 ` Pavan Deolasee <[email protected]>
2017-03-23 18:14 ` Mithun Cy <[email protected]>
2017-03-23 18:35 ` Pavan Deolasee <[email protected]>
2017-03-21 14:19 ` Pavan Deolasee <[email protected]>
2017-03-21 16:21 ` Andres Freund <[email protected]>
2017-03-21 16:49 ` Bruce Momjian <[email protected]>
2017-03-21 17:04 ` Robert Haas <[email protected]>
2017-03-21 17:08 ` Peter Geoghegan <[email protected]>
2017-03-21 17:14 ` Robert Haas <[email protected]>
2017-03-21 17:19 ` Bruce Momjian <[email protected]>
2017-03-21 18:05 ` Petr Jelinek <[email protected]>
2017-03-21 18:56 ` Bruce Momjian <[email protected]>
2017-03-21 18:03 ` Petr Jelinek <[email protected]>
2017-03-21 18:17 ` Robert Haas <[email protected]>
2017-03-21 17:17 ` Bruce Momjian <[email protected]>
2017-03-21 18:15 ` Pavan Deolasee <[email protected]>
2017-03-21 19:00 ` Bruce Momjian <[email protected]>
2017-03-23 18:47 ` Pavan Deolasee <[email protected]>
2017-03-21 18:24 ` Pavan Deolasee <[email protected]>
2017-03-21 19:04 ` Bruce Momjian <[email protected]>
2017-03-21 19:43 ` Alvaro Herrera <[email protected]>
2017-03-21 19:48 ` Bruce Momjian <[email protected]>
2017-03-21 19:56 ` Alvaro Herrera <[email protected]>
2017-03-21 20:04 ` Bruce Momjian <[email protected]>
2017-03-22 12:30 ` Greg Stark <[email protected]>
2017-03-21 16:10 ` Andres Freund <[email protected]>
2017-03-21 13:30 ` Alvaro Herrera <[email protected]>
2017-03-14 01:47 ` Alvaro Herrera <[email protected]>
2017-03-14 18:41 ` Pavan Deolasee <[email protected]>
2017-03-14 19:15 ` Pavan Deolasee <[email protected]>
2017-03-14 19:19 ` Alvaro Herrera <[email protected]>
2017-03-14 19:24 ` Pavan Deolasee <[email protected]>
2017-03-14 20:14 ` Peter Geoghegan <[email protected]>
2017-03-19 07:15 ` Pavan Deolasee <[email protected]>
2017-03-20 16:46 ` Peter Geoghegan <[email protected]>
2017-03-14 19:16 ` Alvaro Herrera <[email protected]>
2017-03-15 19:44 ` Pavan Deolasee <[email protected]>
2017-03-16 12:53 ` Robert Haas <[email protected]>
2017-03-19 07:05 ` Pavan Deolasee <[email protected]>
2017-03-20 14:41 ` Robert Haas <[email protected]>
2017-03-20 15:00 ` Pavan Deolasee <[email protected]>
2017-03-20 15:03 ` Pavan Deolasee <[email protected]>
2017-03-20 15:09 ` Pavan Deolasee <[email protected]>
2017-03-20 16:12 ` Alvaro Herrera <[email protected]>
2017-03-14 17:17 ` Alvaro Herrera <[email protected]>
2017-03-21 13:17 ` Pavan Deolasee <[email protected]>
2017-03-22 10:36 ` Amit Kapila <[email protected]>
2017-03-23 04:58 ` Amit Kapila <[email protected]>
2017-03-23 10:24 ` Pavan Deolasee <[email protected]>
2017-03-23 10:14 ` Pavan Deolasee <[email protected]>
2017-03-23 14:23 ` Amit Kapila <[email protected]>
2017-03-23 18:55 ` Pavan Deolasee <[email protected]>
2020-04-06 03:35 [PATCH v3 05/20] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v2 06/12] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v1 12/19] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v9 10/15] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v4 04/12] Fix docs: "time of the" Justin Pryzby <[email protected]>
2020-04-06 03:35 [PATCH v6 03/10] Fix docs: "time of the" Justin Pryzby <[email protected]>
2024-01-04 22:41 Re: doing also VM cache snapshot and restore with pg_prewarm, having more information of the VM inside PostgreSQL Jim Nasby <[email protected]>
2024-01-05 09:32 ` Re: doing also VM cache snapshot and restore with pg_prewarm, having more information of the VM inside PostgreSQL Cédric Villemain <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox