pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedPATCH: logical_work_mem and logical streaming of large in-progress transactions
536+ messages / 26 participants
[nested] [flat]
* PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-23 04:57 Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 6 replies; 536+ messages in thread
From: Tomas Vondra @ 2017-12-23 04:57 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
Attached is a patch series that implements two features to the logical
replication - ability to define a memory limit for the reorderbuffer
(responsible for building the decoded transactions), and ability to
stream large in-progress transactions (exceeding the memory limit).
I'm submitting those two changes together, because one builds on the
other, and it's beneficial to discuss them together.
PART 1: adding logical_work_mem memory limit (0001)
---------------------------------------------------
Currently, limiting the amount of memory consumed by logical decoding is
tricky (or you might say impossible) for several reasons:
* The value is hard-coded, so it's not quite possible to customize it.
* The amount of decoded changes to keep in memory is restricted by
number of changes. It's not very unclear how this relates to memory
consumption, as the change size depends on table structure, etc.
* The number is "per (sub)transaction", so a transaction with many
subtransactions may easily consume significant amount of memory without
actually hitting the limit.
So the patch does two things. Firstly, it introduces logical_work_mem, a
GUC restricting memory consumed by all transactions currently kept in
the reorder buffer.
Secondly, it adds a simple memory accounting by tracking the amount of
memory used in total (for the whole reorder buffer, to compare against
logical_work_mem) and per transaction (so that we can quickly pick
transaction to spill to disk).
The one wrinkle on the patch is that the memory limit can't be enforced
when reading changes spilled to disk - with multiple subtransactions, we
can't easily predict how many changes to pre-read for each of them. At
that point we still use the existing max_changes_in_memory limit.
Luckily, changes introduced in the other parts of the patch should allow
addressing this deficiency.
PART 2: streaming of large in-progress transactions (0002-0006)
---------------------------------------------------------------
Note: This part is split into multiple smaller chunks, addressing
different parts of the logical decoding infrastructure. That's mostly to
allow easier reviews, though. Ultimately, it's just one patch.
Processing large transactions often results in significant apply lag,
for a couple of reasons. One reason is network bandwidth - while we do
decode the changes incrementally (as we read the WAL), we keep them
locally, either in memory, or spilled to files. Then at commit time, all
the changes get sent to the downstream (and applied) at the same time.
For large transactions the time to do the network transfer may be
significant, causing apply lag.
This patch extends the logical replication infrastructure (output plugin
API, reorder buffer, pgoutput, replication protocol etc.) so allow
streaming of in-progress transactions instead of spilling them to local
files.
The extensions to the API are pretty straightforward. Aside from adding
methods to stream changes/messages and commit a streamed transaction,
the API needs a function to abort a streamed (sub)transaction, and
functions to demarcate a block of streamed changes.
To decode a transaction, we need to know all it's subtransactions, and
invalidations. Currently, those are only known at commit time (although
some assignments may be known earlier), but invalidations are only ever
written in the commit record.
So far that was fine, because we only decode/replay transactions at
commit time, when all of this is known (because it's either in commit
record, or written before it).
But for in-progress transactions (i.e. the subject of interest here),
that is not the case. So the patch modifies WAL-logging to ensure those
two bits of information are written immediately (for wal_level=logical).
For assignments that was fairly simple, thanks to existing caching. For
invalidations, it requires a new WAL record type and a couple of changes
in inval.c.
On the apply side, we simply receive the streamed changes, write them
into a file (one file for toplevel transaction, which is possible thanks
to the assignments being known immediately). And then at commit time the
changes are replayed locally, without having to copy a large chunk of
data over network.
WAL overhead
------------
Of course, these changes to WAL logging are not for free - logging
assignments individually (instead of multiple subtransactions at once)
means higher xlog record overhead. Similarly, (sub)transactions doing a
lot of DDL may result in a lot of invalidations written to WAL (again,
with full xlog record overhead per invalidation).
I've done a number of tests to measure the impact, and for extreme
corner cases the additional amount of WAL is about 40% in both cases.
By an "extreme corner case" I mean a workloads intentionally triggering
many assignments/invalidations, without doing a lot of meaningful work.
For assignments, imagine a single-row table (no indexes), and a
transaction like this one:
BEGIN;
UPDATE t SET v = v + 1;
SAVEPOINT s1;
UPDATE t SET v = v + 1;
SAVEPOINT s2;
UPDATE t SET v = v + 1;
SAVEPOINT s3;
...
UPDATE t SET v = v + 1;
SAVEPOINT s10;
UPDATE t SET v = v + 1;
COMMIT;
For invalidations, add a CREATE TEMPORARY TABLE to each subtransaction.
For more realistic workloads (large table with indexes, runs long enough
to generate FPIs, etc.) the overhead drops below 5%. Which is much more
acceptable, of course, although not perfect.
In both cases, there was pretty much no measurable impact on performance
(as measured by tps).
I do not think there's a way around this requirement (having assignments
and invalidations), if we want to decode in-progress transactions. But
perhaps it would be possible to do some sort of caching (say, at command
level), to reduce the xlog record overhead? Not sure.
All ideas are welcome, of course. In the worst case, I think we can add
a GUC enabling this additional logging - when disabled, streaming of
in-progress transactions would not be possible.
Simplifying ReorderBuffer
-------------------------
One interesting consequence of having assignments is that we could get
rid of the ReorderBuffer iterator, used to merge changes from subxacts.
The assignments allow us to keep changes for each toplevel transaction
in a single list, in LSN order, and just walk it. Abort can be performed
by remembering position of the first change in each subxact, and just
discarding the tail.
This is what the apply worker does with the streamed changes and aborts.
It would also allow us to enforce the memory limit while restoring
transactions spilled to disk, because we would not have the problem with
restoring changes for many subtransactions.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-me.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 10.6K) download
0006-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 13.3K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-23 14:03 Erikjan Rijkers <er@xs4all.nl>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Erikjan Rijkers @ 2017-12-23 14:03 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 2017-12-23 05:57, Tomas Vondra wrote:
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
logical replication of 2 instances is OK but 3 and up fail with:
TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
"reorderbuffer.c", Line: 1773)
I can cobble up a script but I hope you have enough from the assertion
to see what's going wrong...
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-23 20:06 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Erikjan Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-23 20:06 UTC (permalink / raw)
To: Erikjan Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
> On 2017-12-23 05:57, Tomas Vondra wrote:
>> Hi all,
>>
>> Attached is a patch series that implements two features to the logical
>> replication - ability to define a memory limit for the reorderbuffer
>> (responsible for building the decoded transactions), and ability to
>> stream large in-progress transactions (exceeding the memory limit).
>>
>
> logical replication of 2 instances is OK but 3 and up fail with:
>
> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
> "reorderbuffer.c", Line: 1773)
>
> I can cobble up a script but I hope you have enough from the assertion
> to see what's going wrong...
The assertion says that the iterator produces changes in order that does
not correlate with LSN. But I have a hard time understanding how that
could happen, particularly because according to the line number this
happens in ReorderBufferCommit(), i.e. the current (non-streaming) case.
So instructions to reproduce the issue would be very helpful.
Attached is v2 of the patch series, fixing two bugs I discovered today.
I don't think any of these is related to your issue, though.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v2.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch.gz (application/gzip, 10.6K) download
0006-Add-support-for-streaming-to-built-in-replication-v2.patch.gz (application/gzip, 13.9K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-23 22:23 Erik Rijkers <er@xs4all.nl>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-23 22:23 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 2017-12-23 21:06, Tomas Vondra wrote:
> On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
>> On 2017-12-23 05:57, Tomas Vondra wrote:
>>> Hi all,
>>>
>>> Attached is a patch series that implements two features to the
>>> logical
>>> replication - ability to define a memory limit for the reorderbuffer
>>> (responsible for building the decoded transactions), and ability to
>>> stream large in-progress transactions (exceeding the memory limit).
>>>
>>
>> logical replication of 2 instances is OK but 3 and up fail with:
>>
>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>> "reorderbuffer.c", Line: 1773)
>>
>> I can cobble up a script but I hope you have enough from the assertion
>> to see what's going wrong...
>
> The assertion says that the iterator produces changes in order that
> does
> not correlate with LSN. But I have a hard time understanding how that
> could happen, particularly because according to the line number this
> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
> case.
>
> So instructions to reproduce the issue would be very helpful.
Using:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
0006-Add-support-for-streaming-to-built-in-replication-v2.patch
As you expected the problem is the same with these new patches.
I have now tested more, and seen that it not always fails. I guess that
it here fails 3 times out of 4. But the laptop I'm using at the moment
is old and slow -- it may well be a factor as we've seen before [1].
Attached is the bash that I put together. I tested with
NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
often. This same program run with HEAD never seems to fail (I tried a
few dozen times).
thanks,
Erik Rijkers
[1]
https://www.postgresql.org/message-id/3897361c7010c4ac03f358173adbcd60%40xs4all.nl
Attachments:
test.sh (text/x-shellscript, 7.3K) download
#!/bin/bash
unset PGSERVICE PGSERVICEFILE PGDATA PGPORT PGDATABASE
# PGPASSFILE must be set and have the appropriate entries
env | grep ^PG
PROJECT=large_logical
# PROJECT=HEAD
BIN_DIR=$HOME/pg_stuff/pg_installations/pgsql.$PROJECT/bin
POSTGRES=$BIN_DIR/postgres
INITDB=$BIN_DIR/initdb
TMP_DIR=$HOME'/tmp/'$PROJECT
devel_file=${TMP_DIR}'/.devel'
NUM_INSTANCES=3
BASE_PORT=6015 # ports 6015, 6016, 6017
port1=$(( $BASE_PORT + 0 ))
port2=$(( $port1 + 1 ))
port3=$(( $port1 + 2 ))
scale=1 dbname=postgres pubname=pub1 subname=sub1
if [[ ! -d $TMP_DIR ]]; then mkdir $TMP_DIR; fi
echo 's3kr1t' > $devel_file
max_wal_senders=10 # publication side
max_replication_slots=10 # publication side and subscription side
max_worker_processes=12 # subscription side
max_logical_replication_workers=10 # subscription side
max_sync_workers_per_subscription=4 # subscription side
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
data_dir=$TMP_DIR/pgsql.instance${n}/data
server_dir=$TMP_DIR/pgsql.instance${n}
$INITDB --pgdata=$data_dir --encoding=UTF8 --auth=scram-sha-256 --pwfile=$devel_file # --waldir=$xlog_dir
( $POSTGRES -D $data_dir -p $port \
--wal_level=logical \
--max_replication_slots=$max_replication_slots \
--max_worker_processes=$max_worker_processes \
--max_logical_replication_workers=$max_logical_replication_workers \
--max_wal_senders=$max_wal_senders \
--max_sync_workers_per_subscription=$max_sync_workers_per_subscription \
--logging_collector=on \
--log_directory=${server_dir} \
--log_filename=logfile.${port} \
--log_replication_commands=on \
--autovacuum=off & )
# --logical_work_mem=128MB & )
# pg_isready -d $dbname --timeout=60 -p $port
done
#sleep $NUM_INSTANCES
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
num_loop=$(( $NUM_INSTANCES - 1 ))
$BIN_DIR/pgbench --port=$BASE_PORT --quiet --initialize --scale=$scale $dbname
echo "alter table pgbench_history add column hid serial primary key" | $BIN_DIR/psql -d $dbname -p $BASE_PORT -X
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
for n in `seq 1 $num_loop`; do
target_port=$(( $BASE_PORT + $n ))
pg_dump -Fc -p $BASE_PORT \
--exclude-table-data=pgbench_history --exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches --exclude-table-data=pgbench_tellers \
-tpgbench_history -tpgbench_accounts \
-tpgbench_branches -tpgbench_tellers \
$dbname | pg_restore -1 -p $target_port -d $dbname
done
#echo "sleep 2 (after dump/restore)"; sleep 2
for n in `seq 1 $num_loop`; do
pubport=$(( $BASE_PORT + $n - 1 ))
subport=$(( $BASE_PORT + $n ))
appname='casc:'${subport}'<'${pubport}
echo "create publication $pubname for all tables" | psql -d $dbname -p $pubport -X
echo "create subscription $subname
connection 'port=${pubport} dbname=${dbname} application_name=${appname}'
publication $pubname with (enabled=false, slot_name=${subname}_${subport});" | psql -d $dbname -p $subport -X
echo "alter subscription $subname enable; " | psql -d $dbname -p $subport -X
done
c_a1=$( echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port1)
c_b1=$( echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port1)
c_t1=$( echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port1)
c_h1=$( echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port1)
mda1=$( echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdb1=$( echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdt1=$( echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdh1=$( echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
md5_1=$(echo "$mda1 $mdb1 $mdt1 $mdh1" | md5sum | cut -b 1-7 )
printf "a,b,t,h $port1 %6d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1\n" $c_a1 $c_b1 $c_t1 $c_h1
ver1=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port1 )
ver2=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port2 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
ver3=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port3 )
fi
echo
rc=0
while [[ $rc -eq 0 ]]
do
mda2=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdb2=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdt2=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdh2=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
c_a2=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port2)
c_b2=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port2)
c_t2=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port2)
c_h2=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port2)
md5_2=$(echo "$mda2 $mdb2 $mdt2 $mdh2" | md5sum | cut -b 1-7 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
mda3=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdb3=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdt3=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdh3=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
c_a3=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port3)
c_b3=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port3)
c_t3=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port3)
c_h3=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port3)
md5_3=$(echo "$mda3 $mdb3 $mdt3 $mdh3" | md5sum | cut -b 1-7 )
fi
echo "-- $POSTGRES"
printf "a,b,t,h $port1 %7d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1 " $c_a1 $c_b1 $c_t1 $c_h1; echo "$ver1"
printf "a,b,t,h $port2 %7d %6d %6d %6d $mda2 $mdb2 $mdt2 $mdh2 $md5_2 " $c_a2 $c_b2 $c_t2 $c_h2; echo "$ver2"
if [[ $NUM_INSTANCES -gt 2 ]]; then
printf "a,b,t,h $port3 %7d %6d %6d %6d $mda3 $mdb3 $mdt3 $mdh3 $md5_3 " $c_a3 $c_b3 $c_t3 $c_h3; echo "$ver3"
fi
if [[ $NUM_INSTANCES -eq 2 ]]; then
if [[ "$md5_1" == "$md5_2" ]] ; then echo "OK - done."; break; fi
elif [[ $NUM_INSTANCES -eq 3 ]]; then
if [[ "$md5_1" == "$md5_3" ]] ; then echo "OK - done."; break; fi
fi
sleep 1
rc=$?
done
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
data_dir=$TMP_DIR/pgsql.instance${n}/data
$BIN_DIR/pg_ctl stop -w -D $data_dir
done
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-24 00:42 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-24 00:42 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/23/2017 11:23 PM, Erik Rijkers wrote:
> On 2017-12-23 21:06, Tomas Vondra wrote:
>> On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
>>> On 2017-12-23 05:57, Tomas Vondra wrote:
>>>> Hi all,
>>>>
>>>> Attached is a patch series that implements two features to the logical
>>>> replication - ability to define a memory limit for the reorderbuffer
>>>> (responsible for building the decoded transactions), and ability to
>>>> stream large in-progress transactions (exceeding the memory limit).
>>>>
>>>
>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>
>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>> "reorderbuffer.c", Line: 1773)
>>>
>>> I can cobble up a script but I hope you have enough from the assertion
>>> to see what's going wrong...
>>
>> The assertion says that the iterator produces changes in order that does
>> not correlate with LSN. But I have a hard time understanding how that
>> could happen, particularly because according to the line number this
>> happens in ReorderBufferCommit(), i.e. the current (non-streaming) case.
>>
>> So instructions to reproduce the issue would be very helpful.
>
> Using:
>
> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>
> As you expected the problem is the same with these new patches.
>
> I have now tested more, and seen that it not always fails. I guess that
> it here fails 3 times out of 4. But the laptop I'm using at the moment
> is old and slow -- it may well be a factor as we've seen before [1].
>
> Attached is the bash that I put together. I tested with
> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
> often. This same program run with HEAD never seems to fail (I tried a
> few dozen times).
>
Thanks. Unfortunately I still can't reproduce the issue. I even tried
running it in valgrind, to see if there are some memory access issues
(which should also slow it down significantly).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-24 04:51 Craig Ringer <craig@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Craig Ringer @ 2017-12-24 04:51 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 23 December 2017 at 12:57, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
> I'm submitting those two changes together, because one builds on the
> other, and it's beneficial to discuss them together.
>
>
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
>
> Currently, limiting the amount of memory consumed by logical decoding is
> tricky (or you might say impossible) for several reasons:
>
> * The value is hard-coded, so it's not quite possible to customize it.
>
> * The amount of decoded changes to keep in memory is restricted by
> number of changes. It's not very unclear how this relates to memory
> consumption, as the change size depends on table structure, etc.
>
> * The number is "per (sub)transaction", so a transaction with many
> subtransactions may easily consume significant amount of memory without
> actually hitting the limit.
>
Also, even without subtransactions, we assemble a ReorderBufferTXN per
transaction. Since transactions usually occur concurrently, systems with
many concurrent txns can face lots of memory use.
We can't exclude tables that won't actually be replicated at the reorder
buffering phase either. So txns use memory whether or not they do anything
interesting as far as a given logical decoding session is concerned. Even
if we'll throw all the data away we must buffer and assemble it first so we
can make that decision.
Because logical decoding considers snapshots and cid increments even from
other DBs (at least when the txn makes catalog changes) the memory use can
get BIG too. I was recently working with a system that had accumulated 2GB
of snapshots ... on each slot. With 7 slots, one for each DB.
So there's lots of room for difficulty with unpredictable memory use.
So the patch does two things. Firstly, it introduces logical_work_mem, a
> GUC restricting memory consumed by all transactions currently kept in
> the reorder buffer
>
Does this consider the (currently high, IIRC) overhead of tracking
serialized changes?
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-24 09:00 Erik Rijkers <er@xs4all.nl>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-24 09:00 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
>>>>
>>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>>
>>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>>> "reorderbuffer.c", Line: 1773)
>>>>
>>>> I can cobble up a script but I hope you have enough from the
>>>> assertion
>>>> to see what's going wrong...
>>>
>>> The assertion says that the iterator produces changes in order that
>>> does
>>> not correlate with LSN. But I have a hard time understanding how that
>>> could happen, particularly because according to the line number this
>>> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
>>> case.
>>>
>>> So instructions to reproduce the issue would be very helpful.
>>
>> Using:
>>
>> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
>> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
>> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
>> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
>> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
>> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>>
>> As you expected the problem is the same with these new patches.
>>
>> I have now tested more, and seen that it not always fails. I guess
>> that
>> it here fails 3 times out of 4. But the laptop I'm using at the
>> moment
>> is old and slow -- it may well be a factor as we've seen before [1].
>>
>> Attached is the bash that I put together. I tested with
>> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which
>> fails
>> often. This same program run with HEAD never seems to fail (I tried a
>> few dozen times).
>>
>
> Thanks. Unfortunately I still can't reproduce the issue. I even tried
> running it in valgrind, to see if there are some memory access issues
> (which should also slow it down significantly).
One wonders again if 2ndquadrant shouldn't invest in some old hardware
;)
Another Good Thing would be if there was a provision in the buildfarm to
test patches like these.
But I'm probably not to first one to suggest that; no doubt it'll be
possible someday. In the meantime I'll try to repeat this crash on
other machines (but that will be after the holidays).
Erik Rijkers
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-24 13:43 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Craig Ringer <craig@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2017-12-24 13:43 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers
On 12/24/2017 05:51 AM, Craig Ringer wrote:
> On 23 December 2017 at 12:57, Tomas Vondra <tomas.vondra@2ndquadrant.com
> <mailto:tomas.vondra@2ndquadrant.com>> wrote:
>
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
> I'm submitting those two changes together, because one builds on the
> other, and it's beneficial to discuss them together.
>
>
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
>
> Currently, limiting the amount of memory consumed by logical decoding is
> tricky (or you might say impossible) for several reasons:
>
> * The value is hard-coded, so it's not quite possible to customize it.
>
> * The amount of decoded changes to keep in memory is restricted by
> number of changes. It's not very unclear how this relates to memory
> consumption, as the change size depends on table structure, etc.
>
> * The number is "per (sub)transaction", so a transaction with many
> subtransactions may easily consume significant amount of memory without
> actually hitting the limit.
>
>
> Also, even without subtransactions, we assemble a ReorderBufferTXN
> per transaction. Since transactions usually occur concurrently,
> systems with many concurrent txns can face lots of memory use.
>
I don't see how that could be a problem, considering the number of
toplevel transactions is rather limited (to max_connections or so).
> We can't exclude tables that won't actually be replicated at the reorder
> buffering phase either. So txns use memory whether or not they do
> anything interesting as far as a given logical decoding session is
> concerned. Even if we'll throw all the data away we must buffer and
> assemble it first so we can make that decision.
Yep.
> Because logical decoding considers snapshots and cid increments even
> from other DBs (at least when the txn makes catalog changes) the memory
> use can get BIG too. I was recently working with a system that had
> accumulated 2GB of snapshots ... on each slot. With 7 slots, one for
> each DB.
>
> So there's lots of room for difficulty with unpredictable memory use.
>
Yep.
> So the patch does two things. Firstly, it introduces logical_work_mem, a
> GUC restricting memory consumed by all transactions currently kept in
> the reorder buffer
>
>
> Does this consider the (currently high, IIRC) overhead of tracking
> serialized changes?
>
Consider in what sense?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-25 17:40 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-25 17:40 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/24/2017 10:00 AM, Erik Rijkers wrote:
>>>>>
>>>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>>>
>>>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>>>> "reorderbuffer.c", Line: 1773)
>>>>>
>>>>> I can cobble up a script but I hope you have enough from the assertion
>>>>> to see what's going wrong...
>>>>
>>>> The assertion says that the iterator produces changes in order that
>>>> does
>>>> not correlate with LSN. But I have a hard time understanding how that
>>>> could happen, particularly because according to the line number this
>>>> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
>>>> case.
>>>>
>>>> So instructions to reproduce the issue would be very helpful.
>>>
>>> Using:
>>>
>>> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
>>> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
>>> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
>>> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
>>> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
>>> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>>>
>>> As you expected the problem is the same with these new patches.
>>>
>>> I have now tested more, and seen that it not always fails. I guess that
>>> it here fails 3 times out of 4. But the laptop I'm using at the moment
>>> is old and slow -- it may well be a factor as we've seen before [1].
>>>
>>> Attached is the bash that I put together. I tested with
>>> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
>>> often. This same program run with HEAD never seems to fail (I tried a
>>> few dozen times).
>>>
>>
>> Thanks. Unfortunately I still can't reproduce the issue. I even tried
>> running it in valgrind, to see if there are some memory access issues
>> (which should also slow it down significantly).
>
> One wonders again if 2ndquadrant shouldn't invest in some old hardware ;)
>
Well, I've done tests on various machines, including some really slow
ones, and I still haven't managed to reproduce the failures using your
script. So I don't think that would really help. But I have reproduced
it by using a custom stress test script.
Turns out the asserts are overly strict - instead of
Assert(prev_lsn < current_lsn);
it should have been
Assert(prev_lsn <= current_lsn);
because some XLOG records may contain multiple rows (e.g. MULTI_INSERT).
The attached v3 fixes this issue, and also a couple of other thinkos:
1) The AssertChangeLsnOrder assert check was somewhat broken.
2) We've been sending aborts for all subtransactions, even those not yet
streamed. So downstream got confused and fell over because of an assert.
3) The streamed transactions were written to /tmp, using filenames using
subscription OID and XID of the toplevel transaction. That's fine, as
long as there's just a single replica running - if there are more, the
filenames will clash, causing really strange failures. So move the files
to base/pgsql_tmp where regular temporary files are written. I'm not
claiming this is perfect, perhaps we need to invent another location.
FWIW I believe the relation sync cache is somewhat broken by the
streaming. I thought resetting it would be good enough, but it's more
complicated (and trickier) than that. I'm aware of it, and I'll look
into that next - but probably not before 2018.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v3.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v3.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v3.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v3.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v3.patch.gz (application/gzip, 10.7K) download
0006-Add-support-for-streaming-to-built-in-replication-v3.patch.gz (application/gzip, 14.0K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-25 22:08 Erik Rijkers <er@xs4all.nl>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-25 22:08 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
That indeed fixed the problem: running that same pgbench test, I see no
crashes anymore (on any of 3 different machines, and with several
pgbench parameters).
Thank you,
Erik Rijkers
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-26 14:50 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Erik Rijkers <er@xs4all.nl>
0 siblings, 0 replies; 536+ messages in thread
From: Dmitry Dolgov @ 2017-12-26 14:50 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
> On 25 December 2017 at 18:40, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> The attached v3 fixes this issue, and also a couple of other thinkos
Thank you for the patch, it looks quite interesting. After a quick look at
it
(mostly the first one so far, but I'm going to continue) I have a few
questions:
> + * XXX With many subtransactions this might be quite slow, because we'll
have
> + * to walk through all of them. There are some options how we could
improve
> + * that: (a) maintain some secondary structure with transactions sorted
by
> + * amount of changes, (b) not looking for the entirely largest
transaction,
> + * but e.g. for transaction using at least some fraction of the memory
limit,
> + * and (c) evicting multiple transactions at once, e.g. to free a given
portion
> + * of the memory limit (e.g. 50%).
Do you want to address these possible alternatives somehow in this patch or
you
want to left it outside? Maybe it makes sense to apply some combination of
them, e.g. maintain a secondary structure with relatively large
transactions,
and then start evicting them. If it's somehow not enough, then start to
evict
multiple transactions at once (option "c").
> + /*
> + * We clamp manually-set values to at least 64kB. The
maintenance_work_mem
> + * uses a higher minimum value (1MB), so this is OK.
> + */
> + if (*newval < 64)
> + *newval = 64;
> +
I'm not sure what's recommended practice here, but maybe it makes sense to
have a warning here about changing this value to 64kB? Otherwise it can be
unexpected.
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-02 15:07 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-02 15:07 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 12/22/17 23:57, Tomas Vondra wrote:
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
The documentation in this patch contains some references to later
features (streaming). Perhaps that could be separated so that the
patches can be applied independently.
I don't see the need to tie this setting to maintenance_work_mem.
maintenance_work_mem is often set to very large values, which could then
have undesirable side effects on this use.
Moreover, the name logical_work_mem makes it sound like it's a logical
version of work_mem. Maybe we could think of another name.
I think we need a way to report on how much memory is actually used, so
the setting can be tuned. Something analogous to log_temp_files perhaps.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-03 19:53 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-03 19:53 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 01/02/2018 04:07 PM, Peter Eisentraut wrote:
> On 12/22/17 23:57, Tomas Vondra wrote:
>> PART 1: adding logical_work_mem memory limit (0001)
>> ---------------------------------------------------
>
> The documentation in this patch contains some references to later
> features (streaming). Perhaps that could be separated so that the
> patches can be applied independently.
>
Yeah, that's probably a good idea. But now that you mention it, I wonder
if "streaming" is really a good term. We already use it for "streaming
replication" and it may be quite confusing to use it for another feature
(particularly when it's streaming within logical streaming replication).
But I can't really think of a better name ...
> I don't see the need to tie this setting to maintenance_work_mem.
> maintenance_work_mem is often set to very large values, which could
> then have undesirable side effects on this use.
>
Well, we need to pick some default value, and we can either use a fixed
value (not sure what would be a good default) or tie it to an existing
GUC. We only really have work_mem and maintenance_work_mem, and the
walsender process will never use more than one such buffer. Which seems
to be closer to maintenance_work_mem.
Pretty much any default value can have undesirable side effects.
> Moreover, the name logical_work_mem makes it sound like it's a logical
> version of work_mem. Maybe we could think of another name.
>
I won't object to a better name, of course. Any proposals?
> I think we need a way to report on how much memory is actually used,
> so the setting can be tuned. Something analogous to log_temp_files
> perhaps.
>
Yes, I agree. I'm just about to submit an updated version of the patch
series, that also introduces a few columns pg_stat_replication, tracking
this type of stats (amount of data spilled to disk or streamed, etc.).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-03 20:06 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-01-03 20:06 UTC (permalink / raw)
To: pgsql-hackers
Hi,
attached is v4 of the patch series, with a couple of changes:
1) Fixes a bunch of bugs I discovered during stress testing.
I'm not going to go into details, but the main fixes are related to
properly updating progress from the worker, and not streaming when
creating the logical replication slot.
2) Introduces columns into pg_stat_replication.
The new columns track various kinds of statistics (number of xacts,
bytes, ...) about spill-to-disk/streaming. This will be useful when
tuning the GUC memory limit.
3) Two temporary bugfixes that make the patch series work.
The first one (0008) makes sure is_known_subxact is set properly for all
subtransactions, and there's a separate fix in the CF. So this will
eventually go away.
The second one (0009) fixes an issue that is specific to streaming. It
does fix the issue, but I need a bit more time to think about it before
merging it into 0005.
This does pass extensive stress testing with a workload mixing DML, DDL,
subtransactions, aborts, etc. under valgrind. I'm working on extending
the test coverage, and introducing various error conditions (e.g.
walsender/walreceiver timeouts, failures on both ends, etc.).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v4.patch.gz (application/gzip, 9.9K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v4.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v4.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v4.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v4.patch.gz (application/gzip, 10.8K) download
0006-Add-support-for-streaming-to-built-in-replication-v4.patch.gz (application/gzip, 18.2K) download
0007-Track-statistics-for-streaming-spilling-v4.patch.gz (application/gzip, 4.3K) download
0008-BUGFIX-make-sure-subxact-is-marked-as-is_known_as-v4.patch.gz (application/gzip, 571B) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cleanup-v4.patch.gz (application/gzip, 629B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-03 20:13 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-03 20:13 UTC (permalink / raw)
To: pgsql-hackers
On 01/03/2018 09:06 PM, Tomas Vondra wrote:
> Hi,
>
> attached is v4 of the patch series, with a couple of changes:
>
> 1) Fixes a bunch of bugs I discovered during stress testing.
>
> I'm not going to go into details, but the main fixes are related to
> properly updating progress from the worker, and not streaming when
> creating the logical replication slot.
>
> 2) Introduces columns into pg_stat_replication.
>
> The new columns track various kinds of statistics (number of xacts,
> bytes, ...) about spill-to-disk/streaming. This will be useful when
> tuning the GUC memory limit.
>
> 3) Two temporary bugfixes that make the patch series work.
>
Forgot to mention that the v4 also extends the CREATE SUBSCRIPTION to
allow customizing the streaming and memory limit. So you can do
CREATE SUBSCRIPTION ... WITH (streaming=on, work_mem=1024)
and this subscription will allow streaming, and the logica_work_mem (on
provider) will be set to 1MB.
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-09 02:24 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-09 02:24 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 1/3/18 14:53, Tomas Vondra wrote:
>> I don't see the need to tie this setting to maintenance_work_mem.
>> maintenance_work_mem is often set to very large values, which could
>> then have undesirable side effects on this use.
>
> Well, we need to pick some default value, and we can either use a fixed
> value (not sure what would be a good default) or tie it to an existing
> GUC. We only really have work_mem and maintenance_work_mem, and the
> walsender process will never use more than one such buffer. Which seems
> to be closer to maintenance_work_mem.
>
> Pretty much any default value can have undesirable side effects.
Let's just make it an independent setting unless we know any better. We
don't have a lot of settings that depend on other settings, and the ones
we do have a very specific relationship.
>> Moreover, the name logical_work_mem makes it sound like it's a logical
>> version of work_mem. Maybe we could think of another name.
>
> I won't object to a better name, of course. Any proposals?
logical_decoding_[work_]mem?
>> I think we need a way to report on how much memory is actually used,
>> so the setting can be tuned. Something analogous to log_temp_files
>> perhaps.
>
> Yes, I agree. I'm just about to submit an updated version of the patch
> series, that also introduces a few columns pg_stat_replication, tracking
> this type of stats (amount of data spilled to disk or streamed, etc.).
That seems OK. Perhaps we could bring forward the part of that patch
that applies to this feature.
That would also help testing *this* feature and determine what
appropriate settings are.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-09 02:36 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-09 02:36 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 1/3/18 15:13, Tomas Vondra wrote:
> Forgot to mention that the v4 also extends the CREATE SUBSCRIPTION to
> allow customizing the streaming and memory limit. So you can do
>
> CREATE SUBSCRIPTION ... WITH (streaming=on, work_mem=1024)
>
> and this subscription will allow streaming, and the logica_work_mem (on
> provider) will be set to 1MB.
I was wondering already during PG10 development whether we should give
subscriptions a generic configuration array, like databases and roles
have, so we don't have to hardcode a bunch of similar stuff every time
we add an option like this. At the time we only had synchronous_commit,
but now we're adding more.
Also, instead of sticking this into the START_REPLICATION command, could
we just run a SET command? That should work over replication
connections as well.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-11 19:41 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 3 replies; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-11 19:41 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 12/22/17 23:57, Tomas Vondra wrote:
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
>
> Currently, limiting the amount of memory consumed by logical decoding is
> tricky (or you might say impossible) for several reasons:
I would like to see some more discussion on this, but I think not a lot
of people understand the details, so I'll try to write up an explanation
here. This code is also somewhat new to me, so please correct me if
there are inaccuracies, while keeping in mind that I'm trying to simplify.
The data in the WAL is written as it happens, so the changes belonging
to different transactions are all mixed together. One of the jobs of
logical decoding is to reassemble the changes belonging to each
transaction. The top-level data structure for that is the infamous
ReorderBuffer. So as it reads the WAL and sees something about a
transaction, it keeps a copy of that change in memory, indexed by
transaction ID (ReorderBufferChange). When the transaction commits, the
accumulated changes are passed to the output plugin and then freed. If
the transaction aborts, then changes are just thrown away.
So when logical decoding is active, a copy of the changes for each
active transaction is kept in memory (once per walsender).
More precisely, the above happens for each subtransaction. When the
top-level transaction commits, it finds all its subtransactions in the
ReorderBuffer, reassembles everything in the right order, then invokes
the output plugin.
All this could end up using an unbounded amount of memory, so there is a
mechanism to spill changes to disk. The way this currently works is
hardcoded, and this patch proposes to change that.
Currently, when a transaction or subtransaction has accumulated 4096
changes, it is spilled to disk. When the top-level transaction commits,
things are read back from disk to do the final processing mentioned above.
This all works mostly fine, but you can construct some more extreme
cases where this can blow up.
Here is a mundane example. Let's say a change entry takes 100 bytes (it
might contain a new row, or an update key and some new column values,
for example). If you have 100 concurrent active sessions and no
subtransactions, then logical decoding memory is bounded by 4096 * 100 *
100 = 40 MB (per walsender) before things spill to disk.
Now let's say you are using a lot of subtransactions, because you are
using PL functions, exception handling, triggers, doing batch updates.
If you have 200 subtransactions on average per concurrent session, the
memory usage bound in that case would be 4096 * 100 * 100 * 200 = 8 GB
(per walsender). And so on. If you have more concurrent sessions or
larger changes or more subtransactions, you'll use much more than those
8 GB. And if you don't have those 8 GB, then you're stuck at this point.
That is the consideration when we record changes, but we also need
memory when we do the final processing at commit time. That is slightly
less problematic because we only process one top-level transaction at a
time, so the formula is only 4096 * avg_size_of_changes * nr_subxacts
(without the concurrent sessions factor).
So, this patch proposes to improve this as follows:
- We compute the actual size of each ReorderBufferChange and keep a
running tally for each transaction, instead of just counting the number
of changes.
- We have a configuration setting that allows us to change the limit
instead of the hardcoded 4096. The configuration setting is also in
terms of memory, not in number of changes.
- The configuration setting is for the total memory usage per decoding
session, not per subtransaction. (So we also keep a running tally for
the entire ReorderBuffer.)
There are two open issues with this patch:
One, this mechanism only applies when recording changes. The processing
at commit time still uses the previous hardcoded mechanism. The reason
for this is, AFAIU, that as things currently work, you have to have all
subtransactions in memory to do the final processing. There are some
proposals to change this as well, but they are more involved. Arguably,
per my explanation above, memory use at commit time is less likely to be
a problem.
Two, what to do when the memory limit is reached. With the old
accounting, this was easy, because we'd decide for each subtransaction
independently whether to spill it to disk, when it has reached its 4096
limit. Now, we are looking at a global limit, so we have to find a
transaction to spill in some other way. The proposed patch searches
through the entire list of transactions to find the largest one. But as
the patch says:
"XXX With many subtransactions this might be quite slow, because we'll
have to walk through all of them. There are some options how we could
improve that: (a) maintain some secondary structure with transactions
sorted by amount of changes, (b) not looking for the entirely largest
transaction, but e.g. for transaction using at least some fraction of
the memory limit, and (c) evicting multiple transactions at once, e.g.
to free a given portion of the memory limit (e.g. 50%)."
(a) would create more overhead for the case where everything fits into
memory, so it seems unattractive. Some combination of (b) and (c) seems
useful, but we'd have to come up with something concrete.
Thoughts?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-11 23:23 Greg Stark <stark@mit.edu>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2 siblings, 1 reply; 536+ messages in thread
From: Greg Stark @ 2018-01-11 23:23 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 11 January 2018 at 19:41, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> Two, what to do when the memory limit is reached. With the old
> accounting, this was easy, because we'd decide for each subtransaction
> independently whether to spill it to disk, when it has reached its 4096
> limit. Now, we are looking at a global limit, so we have to find a
> transaction to spill in some other way. The proposed patch searches
> through the entire list of transactions to find the largest one. But as
> the patch says:
>
> "XXX With many subtransactions this might be quite slow, because we'll
> have to walk through all of them. There are some options how we could
> improve that: (a) maintain some secondary structure with transactions
> sorted by amount of changes, (b) not looking for the entirely largest
> transaction, but e.g. for transaction using at least some fraction of
> the memory limit, and (c) evicting multiple transactions at once, e.g.
> to free a given portion of the memory limit (e.g. 50%)."
AIUI spilling to disk doesn't affect absorbing future updates, we
would just keep accumulating them in memory right? We won't need to
unspill until it comes time to commit.
Is there any actual advantage to picking the largest transaction? it
means fewer spills and fewer unspills at commit time but that just a
bigger spike of i/o and more of a chance of spilling more than
necessary to get by. In the end it'll be more or less the same amount
of data read back, just all in one big spike when spilling and one big
spike when committing. If you spilled smaller transactions you would
have a small amount of i/o more frequently and have to read back small
amounts for many commits. But it would add up to the same amount of
i/o (or less if you avoid spilling more than necessary).
The real aim should be to try to pick the transaction that will be
committed furthest in the future. That gives you the most memory to
use for live transactions for the longest time and could let you
process the maximum amount of transactions without spilling them. So
either the oldest transaction (in the expectation that it's been open
a while and appears to be a long-lived batch job that will stay open
for a long time) or the youngest transaction (in the expectation that
all transactions are more or less equally long-lived) might make
sense.
--
greg
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-12 16:35 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Greg Stark <stark@mit.edu>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-12 16:35 UTC (permalink / raw)
To: Greg Stark <stark@mit.edu>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 1/11/18 18:23, Greg Stark wrote:
> AIUI spilling to disk doesn't affect absorbing future updates, we
> would just keep accumulating them in memory right? We won't need to
> unspill until it comes time to commit.
Once a transaction has been serialized, future updates keep accumulating
in memory, until perhaps it gets serialized again. But then at commit
time, if a transaction has been partially serialized at all, all the
remaining changes are also serialized before the whole thing is read
back in (see reorderbuffer.c line 855).
So one optimization would be to specially keep track of all transactions
that have been serialized already and pick those first for further
serialization, because it will be done eventually anyway.
But this is only a secondary optimization, because it doesn't help in
the extreme cases that either no (or few) transactions have been
serialized or all (or most) transactions have been serialized.
> The real aim should be to try to pick the transaction that will be
> committed furthest in the future. That gives you the most memory to
> use for live transactions for the longest time and could let you
> process the maximum amount of transactions without spilling them. So
> either the oldest transaction (in the expectation that it's been open
> a while and appears to be a long-lived batch job that will stay open
> for a long time) or the youngest transaction (in the expectation that
> all transactions are more or less equally long-lived) might make
> sense.
Yes, that makes sense. We'd still need to keep a separate ordered list
of transactions somewhere, but that might be easier if we just order
them in the order we see them.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-13 04:01 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-01-13 04:01 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 01/11/2018 08:41 PM, Peter Eisentraut wrote:
> On 12/22/17 23:57, Tomas Vondra wrote:
>> PART 1: adding logical_work_mem memory limit (0001)
>> ---------------------------------------------------
>>
>> Currently, limiting the amount of memory consumed by logical decoding is
>> tricky (or you might say impossible) for several reasons:
>
> I would like to see some more discussion on this, but I think not a lot
> of people understand the details, so I'll try to write up an explanation
> here. This code is also somewhat new to me, so please correct me if
> there are inaccuracies, while keeping in mind that I'm trying to simplify.
>
> ... snip ...
Thanks for a comprehensive summary of the patch!
>
> "XXX With many subtransactions this might be quite slow, because we'll
> have to walk through all of them. There are some options how we could
> improve that: (a) maintain some secondary structure with transactions
> sorted by amount of changes, (b) not looking for the entirely largest
> transaction, but e.g. for transaction using at least some fraction of
> the memory limit, and (c) evicting multiple transactions at once, e.g.
> to free a given portion of the memory limit (e.g. 50%)."
>
> (a) would create more overhead for the case where everything fits into
> memory, so it seems unattractive. Some combination of (b) and (c) seems
> useful, but we'd have to come up with something concrete.
>
Yeah, when writing that comment I was worried that (a) might get rather
expensive. I was thinking about maintaining a dlist of transactions
sorted by size (ReorderBuffer now only has a hash table), so that we
could evict transactions from the beginning of the list.
But while that speeds up the choice of transactions to evict, the added
cost is rather high, particularly when most transactions are roughly of
the same size. Because in that case we probably have to move the nodes
around in the list quite often. So it seems wiser to just walk the list
once when looking for a victim.
What I'm thinking about instead is tracking just some approximated
version of this - it does not really matter whether we evict the really
largest transaction or one that is a couple of kilobytes smaller. What
we care about is an answer to this question:
Is there some very large transaction that we could evict to free
a lot of memory, or are all transactions fairly small?
So perhaps we can define some "size classes" and track to which of them
each transaction belongs. For example, we could split the memory limit
into 100 buckets, each representing a 1% size increment.
A transaction would not switch the class very often, and it would be
trivial to pick the largest transaction. When all the transactions are
squashed in the smallest classes, we may switch to some alternative
strategy. Not sure.
In any case, I don't really know how expensive the selection actually
is, and if it's an issue. I'll do some measurements.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-13 04:19 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-13 04:19 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Greg Stark <stark@mit.edu>; +Cc: pgsql-hackers
On 01/12/2018 05:35 PM, Peter Eisentraut wrote:
> On 1/11/18 18:23, Greg Stark wrote:
>> AIUI spilling to disk doesn't affect absorbing future updates, we
>> would just keep accumulating them in memory right? We won't need to
>> unspill until it comes time to commit.
>
> Once a transaction has been serialized, future updates keep accumulating
> in memory, until perhaps it gets serialized again. But then at commit
> time, if a transaction has been partially serialized at all, all the
> remaining changes are also serialized before the whole thing is read
> back in (see reorderbuffer.c line 855).
>
> So one optimization would be to specially keep track of all transactions
> that have been serialized already and pick those first for further
> serialization, because it will be done eventually anyway.
>
> But this is only a secondary optimization, because it doesn't help in
> the extreme cases that either no (or few) transactions have been
> serialized or all (or most) transactions have been serialized.
>
>> The real aim should be to try to pick the transaction that will be
>> committed furthest in the future. That gives you the most memory to
>> use for live transactions for the longest time and could let you
>> process the maximum amount of transactions without spilling them. So
>> either the oldest transaction (in the expectation that it's been open
>> a while and appears to be a long-lived batch job that will stay open
>> for a long time) or the youngest transaction (in the expectation that
>> all transactions are more or less equally long-lived) might make
>> sense.
>
> Yes, that makes sense. We'd still need to keep a separate ordered list
> of transactions somewhere, but that might be easier if we just order
> them in the order we see them.
>
Wouldn't the 'toplevel_by_lsn' be suitable for this? Subtransactions
don't really commit independently, but as part of the toplevel xact. And
that list is ordered by LSN, which is pretty much exactly the order in
which we see the transactions.
I feel somewhat uncomfortable about evicting oldest (or youngest)
transactions for based on some assumed correlation with the commit
order. I'm pretty sure that will bite us badly for some workloads.
Another somewhat non-intuitive detail is that because ReorderBuffer
switched to Generation allocator for changes (which usually represent
99% of the memory used during decoding), it does not reuse memory the
way AllocSet does. Actually, it does not reuse memory at all, aiming to
eventually give the memory back to libc (which AllocSet can't do).
Because of this evicting the youngest transactions seems like a quite
bad idea, because those chunks will not be reused and there may be other
chunks on the blocks, preventing their release.
Yeah, complicated stuff.
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-18 13:24 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-18 13:24 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Greg Stark <stark@mit.edu>; +Cc: pgsql-hackers
On 1/12/18 23:19, Tomas Vondra wrote:
> Wouldn't the 'toplevel_by_lsn' be suitable for this? Subtransactions
> don't really commit independently, but as part of the toplevel xact. And
> that list is ordered by LSN, which is pretty much exactly the order in
> which we see the transactions.
Yes indeed. There is even ReorderBufferGetOldestTXN().
> Another somewhat non-intuitive detail is that because ReorderBuffer
> switched to Generation allocator for changes (which usually represent
> 99% of the memory used during decoding), it does not reuse memory the
> way AllocSet does. Actually, it does not reuse memory at all, aiming to
> eventually give the memory back to libc (which AllocSet can't do).
>
> Because of this evicting the youngest transactions seems like a quite
> bad idea, because those chunks will not be reused and there may be other
> chunks on the blocks, preventing their release.
Right. But this raises the question whether we are doing the memory
accounting on the right level. If we are doing all this tracking based
on ReorderBufferChanges, but then serializing changes possibly doesn't
actually free any memory in the operating system, that's no good. Can
we get some usage statistics out of the memory context? It seems like
we need to keep serializing transactions until we actually see the
memory context size drop.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-19 14:34 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-19 14:34 UTC (permalink / raw)
To: pgsql-hackers
Attached is v5, fixing a silly bug in part 0006, causing segfault when
creating a subscription.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v5.patch.gz (application/gzip, 9.9K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v5.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v5.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v5.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v5.patch.gz (application/gzip, 10.8K) download
0006-Add-support-for-streaming-to-built-in-replication-v5.patch.gz (application/gzip, 18.3K) download
0007-Track-statistics-for-streaming-spilling-v5.patch.gz (application/gzip, 4.3K) download
0008-BUGFIX-make-sure-subxact-is-marked-as-is_known_as-v5.patch.gz (application/gzip, 570B) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cleanup-v5.patch.gz (application/gzip, 630B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-19 22:08 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-19 22:08 UTC (permalink / raw)
To: pgsql-hackers
On 01/19/2018 03:34 PM, Tomas Vondra wrote:
> Attached is v5, fixing a silly bug in part 0006, causing segfault when
> creating a subscription.
>
Meh, there was a bug in the sgml docs (<variable> vs. <varname>),
causing another failure. Hopefully v6 will pass the CI build, it does
pass a build with the same parameters on my system.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v6.patch.gz (application/gzip, 9.9K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v6.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v6.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v6.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v6.patch.gz (application/gzip, 10.8K) download
0006-Add-support-for-streaming-to-built-in-replication-v6.patch.gz (application/gzip, 18.3K) download
0007-Track-statistics-for-streaming-spilling-v6.patch.gz (application/gzip, 4.3K) download
0008-BUGFIX-make-sure-subxact-is-marked-as-is_known_as-v6.patch.gz (application/gzip, 570B) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cleanup-v6.patch.gz (application/gzip, 631B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-01-31 06:53 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Masahiko Sawada @ 2018-01-31 06:53 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On Sat, Jan 20, 2018 at 7:08 AM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> On 01/19/2018 03:34 PM, Tomas Vondra wrote:
>> Attached is v5, fixing a silly bug in part 0006, causing segfault when
>> creating a subscription.
>>
>
> Meh, there was a bug in the sgml docs (<variable> vs. <varname>),
> causing another failure. Hopefully v6 will pass the CI build, it does
> pass a build with the same parameters on my system.
Thank you for working on this. This patch would be helpful for
synchronous replication.
I haven't looked at the code deeply yet, but I've reviewed the v6
patch set especially on subscriber side. All of the patches can be
applied to current HEAD cleanly. Here is review comment.
----
CREATE SUBSCRIPTION commands accept work_mem < 64 but it leads ERROR
on publisher side when starting replication. Probably we should check
the value on the subscriber side as well.
----
When streaming = on, if we drop subscription in the middle of
receiving stream changes, DROP SUBSCRIPTION could leak tmp files
(.chages file and .subxacts file). Also it also happens when a
transaction on upstream aborted without abort record.
----
Since we can change both streaming option and work_mem option by ALTER
SUBSCRIPTION, documentation of ALTER SUBSCRIPTION needs to be updated.
----
If we create a subscription without any options, both
pg_subscription.substream and pg_subscription.subworkmem are set to
null. However, since GetSubscription are't aware of NULL we start the
replication with invalid options like follows.
LOG: received replication command: START_REPLICATION SLOT "hoge_sub"
LOGICAL 0/0 (proto_version '2', work_mem '893219954', streaming 'on',
publication_names '"hoge_pub"')
I think we can set substream to false and subworkmem to -1 instead of
null, and then makes libpqrcv_startstreaming not set streaming option
if stream is -1.
----
Some WARNING messages appeared. Maybe these are for debug purpose?
WARNING: updating stream stats 0x1c12ef8 4 3 65604
WARNING: UpdateSpillStats: updating stats 0x1c12ef8 0 0 0 39 41 2632080
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-02-01 14:51 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-02-01 14:51 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
To close out this commit fest, I'm setting both of these patches as
returned with feedback, as there are apparently significant issues to be
addressed. Feel free to move them to the next commit fest when you
think they are ready to be continued.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-02-01 22:50 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-02-01 22:50 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
On 01/31/2018 07:53 AM, Masahiko Sawada wrote:
> On Sat, Jan 20, 2018 at 7:08 AM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>> On 01/19/2018 03:34 PM, Tomas Vondra wrote:
>>> Attached is v5, fixing a silly bug in part 0006, causing segfault when
>>> creating a subscription.
>>>
>>
>> Meh, there was a bug in the sgml docs (<variable> vs. <varname>),
>> causing another failure. Hopefully v6 will pass the CI build, it does
>> pass a build with the same parameters on my system.
>
> Thank you for working on this. This patch would be helpful for
> synchronous replication.
>
> I haven't looked at the code deeply yet, but I've reviewed the v6
> patch set especially on subscriber side. All of the patches can be
> applied to current HEAD cleanly. Here is review comment.
>
> ----
> CREATE SUBSCRIPTION commands accept work_mem < 64 but it leads ERROR
> on publisher side when starting replication. Probably we should check
> the value on the subscriber side as well.
>
> ----
> When streaming = on, if we drop subscription in the middle of
> receiving stream changes, DROP SUBSCRIPTION could leak tmp files
> (.chages file and .subxacts file). Also it also happens when a
> transaction on upstream aborted without abort record.
>
> ----
> Since we can change both streaming option and work_mem option by ALTER
> SUBSCRIPTION, documentation of ALTER SUBSCRIPTION needs to be updated.
>
> ----
> If we create a subscription without any options, both
> pg_subscription.substream and pg_subscription.subworkmem are set to
> null. However, since GetSubscription are't aware of NULL we start the
> replication with invalid options like follows.
> LOG: received replication command: START_REPLICATION SLOT "hoge_sub"
> LOGICAL 0/0 (proto_version '2', work_mem '893219954', streaming 'on',
> publication_names '"hoge_pub"')
>
> I think we can set substream to false and subworkmem to -1 instead of
> null, and then makes libpqrcv_startstreaming not set streaming option
> if stream is -1.
>
> ----
> Some WARNING messages appeared. Maybe these are for debug purpose?
>
> WARNING: updating stream stats 0x1c12ef8 4 3 65604
> WARNING: UpdateSpillStats: updating stats 0x1c12ef8 0 0 0 39 41 2632080
>
> Regards,
>
Thanks for the review! I'll address the issues in the next version of
the patch.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-02-01 22:51 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-02-01 22:51 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: pgsql-hackers
On 02/01/2018 03:51 PM, Peter Eisentraut wrote:
> To close out this commit fest, I'm setting both of these patches as
> returned with feedback, as there are apparently significant issues to be
> addressed. Feel free to move them to the next commit fest when you
> think they are ready to be continued.
>
Will do. Thanks for the feedback.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 01:12 Andres Freund <andres@anarazel.de>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Andres Freund @ 2018-03-02 01:12 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2018-02-01 23:51:55 +0100, Tomas Vondra wrote:
> On 02/01/2018 03:51 PM, Peter Eisentraut wrote:
> > To close out this commit fest, I'm setting both of these patches as
> > returned with feedback, as there are apparently significant issues to be
> > addressed. Feel free to move them to the next commit fest when you
> > think they are ready to be continued.
> >
>
> Will do. Thanks for the feedback.
Hm, this CF entry is marked as needs review as of 2018-03-01 12:54:48,
but I don't see a newer version posted?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 02:33 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-03-02 02:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 03/02/2018 02:12 AM, Andres Freund wrote:
> On 2018-02-01 23:51:55 +0100, Tomas Vondra wrote:
>> On 02/01/2018 03:51 PM, Peter Eisentraut wrote:
>>> To close out this commit fest, I'm setting both of these patches as
>>> returned with feedback, as there are apparently significant issues to be
>>> addressed. Feel free to move them to the next commit fest when you
>>> think they are ready to be continued.
>>>
>>
>> Will do. Thanks for the feedback.
>
> Hm, this CF entry is marked as needs review as of 2018-03-01 12:54:48,
> but I don't see a newer version posted?
>
Ah, apologies - that's due to moving the patch from the last CF (it was
marked as RWF so I had to reopen it before moving it). I'll submit a new
version of the patch shortly, please mark it as WOA until then.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 02:39 David Steele <david@pgmasters.net>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: David Steele @ 2018-03-02 02:39 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Hi Tomas.
On 3/1/18 9:33 PM, Tomas Vondra wrote:
> On 03/02/2018 02:12 AM, Andres Freund wrote:
>> On 2018-02-01 23:51:55 +0100, Tomas Vondra wrote:
>>> On 02/01/2018 03:51 PM, Peter Eisentraut wrote:
>>>> To close out this commit fest, I'm setting both of these patches as
>>>> returned with feedback, as there are apparently significant issues to be
>>>> addressed. Feel free to move them to the next commit fest when you
>>>> think they are ready to be continued.
>>>>
>>>
>>> Will do. Thanks for the feedback.
>>
>> Hm, this CF entry is marked as needs review as of 2018-03-01 12:54:48,
>> but I don't see a newer version posted?
>>
>
> Ah, apologies - that's due to moving the patch from the last CF (it was
> marked as RWF so I had to reopen it before moving it). I'll submit a new
> version of the patch shortly, please mark it as WOA until then.
Marked as Waiting on Author.
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 20:05 Andres Freund <andres@anarazel.de>
parent: David Steele <david@pgmasters.net>
0 siblings, 1 reply; 536+ messages in thread
From: Andres Freund @ 2018-03-02 20:05 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Hi,
On 2018-03-01 21:39:36 -0500, David Steele wrote:
> On 3/1/18 9:33 PM, Tomas Vondra wrote:
> > On 03/02/2018 02:12 AM, Andres Freund wrote:
> > > Hm, this CF entry is marked as needs review as of 2018-03-01 12:54:48,
> > > but I don't see a newer version posted?
> > >
> >
> > Ah, apologies - that's due to moving the patch from the last CF (it was
> > marked as RWF so I had to reopen it before moving it). I'll submit a new
> > version of the patch shortly, please mark it as WOA until then.
>
> Marked as Waiting on Author.
Sorry to be the hard-ass, but given this patch hasn't been moved forward
since 2018-01-19, I'm not sure why it's elegible to be in this CF in the
first place?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 20:06 Robert Haas <robertmhaas@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Robert Haas @ 2018-03-02 20:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Mar 1, 2018 at 9:33 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Ah, apologies - that's due to moving the patch from the last CF (it was
> marked as RWF so I had to reopen it before moving it). I'll submit a new
> version of the patch shortly, please mark it as WOA until then.
So, the way it's supposed to work is you resubmit the patch first and
then re-activate the CF entry. If you get to re-activate the CF entry
without actually updating the patch, and then submit the patch
afterwards, then the CF deadline becomes largely meaningless. I think
a new patch should rejected as untimely.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 20:21 David Steele <david@pgmasters.net>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: David Steele @ 2018-03-02 20:21 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 3/2/18 3:06 PM, Robert Haas wrote:
> On Thu, Mar 1, 2018 at 9:33 PM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>> Ah, apologies - that's due to moving the patch from the last CF (it was
>> marked as RWF so I had to reopen it before moving it). I'll submit a new
>> version of the patch shortly, please mark it as WOA until then.
>
> So, the way it's supposed to work is you resubmit the patch first and
> then re-activate the CF entry. If you get to re-activate the CF entry
> without actually updating the patch, and then submit the patch
> afterwards, then the CF deadline becomes largely meaningless. I think
> a new patch should rejected as untimely.
Hmmm, I missed that implication last night. I'll mark this Returned
with Feedback.
Tomas, please move to the next CF once you have an updated patch.
Thanks,
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-02 21:06 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: David Steele <david@pgmasters.net>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-03-02 21:06 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 03/02/2018 09:21 PM, David Steele wrote:
> On 3/2/18 3:06 PM, Robert Haas wrote:
>> On Thu, Mar 1, 2018 at 9:33 PM, Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>>> Ah, apologies - that's due to moving the patch from the last CF (it was
>>> marked as RWF so I had to reopen it before moving it). I'll submit a new
>>> version of the patch shortly, please mark it as WOA until then.
>>
>> So, the way it's supposed to work is you resubmit the patch first and
>> then re-activate the CF entry. If you get to re-activate the CF entry
>> without actually updating the patch, and then submit the patch
>> afterwards, then the CF deadline becomes largely meaningless. I think
>> a new patch should rejected as untimely.
>
> Hmmm, I missed that implication last night. I'll mark this Returned
> with Feedback.
>
> Tomas, please move to the next CF once you have an updated patch.
>
Can you guys please point me to the CF rules that say this? Because my
understanding (and not just mine, AFAICS) was obviously different.
Clearly there's a disconnect somewhere.
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 00:55 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-03-03 00:55 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers
Hi there,
attached is an updated patch fixing all the reported issues (a bit more
about those below).
The main change in this patch version is reworked logging of subxact
assignments, which needs to be done immediately for incremental decoding
to work properly.
The previous patch versions did that by logging a separate xlog record,
which however had rather noticeable space overhead (~40% on a worst-case
test - tiny table, no FPWs, ...). While in practice the overhead would
be much closer to 0%, it still seemed unacceptable.
Andres proposed doing something like we do with replication origins in
XLogRecordAssemble, i.e. inventing a special block, and embedding the
assignment info into that (in the next xlog record). This turned out to
be working quite well, and the worst-case space overhead dropped to ~5%.
I have attempted to do something like that with the invalidations, which
is the other thing that needs to be logged immediately for incremental
decoding to work correctly. The plan was to use the same approach as for
assignments, i.e. embed the invalidations into the next xlog record and
stop sending them in the commit message. That however turned out to be
much more complicated - the embedding is fairly trivial, of course, but
unlike assignments the invalidations are needed for hot standbys. If we
only send them incrementally, I think the standby would have to collect
from the WAL records, and store them in a way that survives restarts.
So for invalidations the patch uses the original approach with a new
type xlog record type (ignored by standby), and still logging the
invalidations in commit record (which is that the standby relies on).
On 02/01/2018 11:50 PM, Tomas Vondra wrote:
> On 01/31/2018 07:53 AM, Masahiko Sawada wrote:
> ...
>> ----
>> CREATE SUBSCRIPTION commands accept work_mem < 64 but it leads ERROR
>> on publisher side when starting replication. Probably we should check
>> the value on the subscriber side as well.
>>
Added.
>> ----
>> When streaming = on, if we drop subscription in the middle of
>> receiving stream changes, DROP SUBSCRIPTION could leak tmp files
>> (.chages file and .subxacts file). Also it also happens when a
>> transaction on upstream aborted without abort record.
>>
Right. The files would get cleaned up eventually during restart (just
like other temporary files), but leaking them after DROP SUBSCRIPTION is
not cool. So I've added a simple tracking of files (or rather streamed
XIDs) in the worker, and clean them explicitly on exit.
>> ----
>> Since we can change both streaming option and work_mem option by ALTER
>> SUBSCRIPTION, documentation of ALTER SUBSCRIPTION needs to be updated.
>>
Yep, I've added note that work_mem and streaming can also be changed.
Those changes won't be applied to the already running worker, though.
>> ----
>> If we create a subscription without any options, both
>> pg_subscription.substream and pg_subscription.subworkmem are set to
>> null. However, since GetSubscription are't aware of NULL we start the
>> replication with invalid options like follows.
>> LOG: received replication command: START_REPLICATION SLOT "hoge_sub"
>> LOGICAL 0/0 (proto_version '2', work_mem '893219954', streaming 'on',
>> publication_names '"hoge_pub"')
>>
>> I think we can set substream to false and subworkmem to -1 instead of
>> null, and then makes libpqrcv_startstreaming not set streaming option
>> if stream is -1.
>>
Good catch! I've done pretty much what you suggested here, i.e. store
-1/false instead and then handle that in libpqrcv_startstreaming.
>> ----
>> Some WARNING messages appeared. Maybe these are for debug purpose?
>>
>> WARNING: updating stream stats 0x1c12ef8 4 3 65604
>> WARNING: UpdateSpillStats: updating stats 0x1c12ef8 0 0 0 39 41 2632080
>>
Yeah, those should be removed.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer.patch.gz (application/gzip, 9.9K) download
0002-Immediatel-WAL-log-assignments.patch.gz (application/gzip, 6.7K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 10.8K) download
0006-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 19.2K) download
0007-Track-statistics-for-streaming-spilling.patch.gz (application/gzip, 4.3K) download
0008-BUGFIX-make-sure-subxact-is-marked-as-is_known_as_su.patch.gz (application/gzip, 569B) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch.gz (application/gzip, 627B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:00 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-03-03 01:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; David Steele <david@pgmasters.net>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 03/02/2018 09:05 PM, Andres Freund wrote:
> Hi,
>
> On 2018-03-01 21:39:36 -0500, David Steele wrote:
>> On 3/1/18 9:33 PM, Tomas Vondra wrote:
>>> On 03/02/2018 02:12 AM, Andres Freund wrote:
>>>> Hm, this CF entry is marked as needs review as of 2018-03-01 12:54:48,
>>>> but I don't see a newer version posted?
>>>>
>>>
>>> Ah, apologies - that's due to moving the patch from the last CF (it was
>>> marked as RWF so I had to reopen it before moving it). I'll submit a new
>>> version of the patch shortly, please mark it as WOA until then.
>>
>> Marked as Waiting on Author.
>
> Sorry to be the hard-ass, but given this patch hasn't been moved forward
> since 2018-01-19, I'm not sure why it's elegible to be in this CF in the
> first place?
>
That is somewhat misleading, I think. You're right the last version was
submitted on 2018-01-19, but the next review arrived on 2018-01-31, i.e.
right at the end of the CF. So it's not like the patch was sitting there
with unresolved issues. Based on that review the patch was marked as RWF
and thus not moved to 2018-03 automatically.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:01 Andres Freund <andres@anarazel.de>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Andres Freund @ 2018-03-03 01:01 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
> That is somewhat misleading, I think. You're right the last version was
> submitted on 2018-01-19, but the next review arrived on 2018-01-31, i.e.
> right at the end of the CF. So it's not like the patch was sitting there
> with unresolved issues. Based on that review the patch was marked as RWF
> and thus not moved to 2018-03 automatically.
I don't see how this changes anything.
- Andres
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:34 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-03-03 01:34 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: David Steele <david@pgmasters.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 03/03/2018 02:01 AM, Andres Freund wrote:
> On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
>> That is somewhat misleading, I think. You're right the last version
>> was submitted on 2018-01-19, but the next review arrived on
>> 2018-01-31, i.e. right at the end of the CF. So it's not like the
>> patch was sitting there with unresolved issues. Based on that
>> review the patch was marked as RWF and thus not moved to 2018-03
>> automatically.
>
> I don't see how this changes anything.
>
You've used "The patch hasn't moved forward since 2018-01-19," as an
argument why the patch is not eligible for 2018-03. I suggest that
argument is misleading, because patches generally do not move without
reviews, and it's difficult to respond to a review that arrives on the
last day of a commitfest.
Consider that without the review, the patch would end up with NR status,
and would be moved to the next CF automatically. Isn't that a bit weird?
kind regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:36 Andres Freund <andres@anarazel.de>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Andres Freund @ 2018-03-03 01:36 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2018-03-03 02:34:06 +0100, Tomas Vondra wrote:
> On 03/03/2018 02:01 AM, Andres Freund wrote:
> > On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
> >> That is somewhat misleading, I think. You're right the last version
> >> was submitted on 2018-01-19, but the next review arrived on
> >> 2018-01-31, i.e. right at the end of the CF. So it's not like the
> >> patch was sitting there with unresolved issues. Based on that
> >> review the patch was marked as RWF and thus not moved to 2018-03
> >> automatically.
> >
> > I don't see how this changes anything.
> >
>
> You've used "The patch hasn't moved forward since 2018-01-19," as an
> argument why the patch is not eligible for 2018-03. I suggest that
> argument is misleading, because patches generally do not move without
> reviews, and it's difficult to respond to a review that arrives on the
> last day of a commitfest.
>
> Consider that without the review, the patch would end up with NR status,
> and would be moved to the next CF automatically. Isn't that a bit weird?
Not sure I follow. The point is that nobody would have complained if
you'd moved the patch into this fest if you'd updated it *before* it
started?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:37 David Steele <david@pgmasters.net>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 536+ messages in thread
From: David Steele @ 2018-03-03 01:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 3/2/18 8:01 PM, Andres Freund wrote:
> On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
>> That is somewhat misleading, I think. You're right the last version was
>> submitted on 2018-01-19, but the next review arrived on 2018-01-31, i.e.
>> right at the end of the CF. So it's not like the patch was sitting there
>> with unresolved issues. Based on that review the patch was marked as RWF
>> and thus not moved to 2018-03 automatically.
>
> I don't see how this changes anything.
I agree that things could be clearer, and Andres has produced a great
document that we can build on. The old one had gotten a bit stale.
However, I think it's pretty obvious that a CF entry should be
accompanied with a patch. It sounds like the timing was awkward but you
still had 28 days to produce a new patch.
I also notice that you submitted 7 patches in this CF but are reviewing
zero.
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 01:54 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: David Steele <david@pgmasters.net>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-03-03 01:54 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 03/03/2018 02:37 AM, David Steele wrote:
> On 3/2/18 8:01 PM, Andres Freund wrote:
>> On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
>>> That is somewhat misleading, I think. You're right the last version was
>>> submitted on 2018-01-19, but the next review arrived on 2018-01-31, i.e.
>>> right at the end of the CF. So it's not like the patch was sitting there
>>> with unresolved issues. Based on that review the patch was marked as RWF
>>> and thus not moved to 2018-03 automatically.
>>
>> I don't see how this changes anything.
>
> I agree that things could be clearer, and Andres has produced a great
> document that we can build on. The old one had gotten a bit stale.
>
> However, I think it's pretty obvious that a CF entry should be
> accompanied with a patch. It sounds like the timing was awkward but
> you still had 28 days to produce a new patch.
>
Based on internal discussion I'm not so sure about the "pretty obvious"
part. It certainly wasn't that obvious to me, otherwise I'd submit the
revised patch earlier - hindsight is 20/20.
> I also notice that you submitted 7 patches in this CF but are
> reviewing zero.
>
I've volunteered to review a couple of patches at the FOSDEM Developer
Meeting - I thought Stephen was entering that into the CF app, not sure
where it got lost.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 02:08 David Steele <david@pgmasters.net>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: David Steele @ 2018-03-03 02:08 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 3/2/18 8:54 PM, Tomas Vondra wrote:
> On 03/03/2018 02:37 AM, David Steele wrote:
>> On 3/2/18 8:01 PM, Andres Freund wrote:
>>> On 2018-03-03 02:00:46 +0100, Tomas Vondra wrote:
>>>> That is somewhat misleading, I think. You're right the last version was
>>>> submitted on 2018-01-19, but the next review arrived on 2018-01-31, i.e.
>>>> right at the end of the CF. So it's not like the patch was sitting there
>>>> with unresolved issues. Based on that review the patch was marked as RWF
>>>> and thus not moved to 2018-03 automatically.
>>>
>>> I don't see how this changes anything.
>>
>> I agree that things could be clearer, and Andres has produced a great
>> document that we can build on. The old one had gotten a bit stale.
>>
>> However, I think it's pretty obvious that a CF entry should be
>> accompanied with a patch. It sounds like the timing was awkward but
>> you still had 28 days to produce a new patch.
>
> Based on internal discussion I'm not so sure about the "pretty obvious"
> part. It certainly wasn't that obvious to me, otherwise I'd submit the
> revised patch earlier - hindsight is 20/20.
Indeed it is. Be assured that nobody takes pleasure in pushing patches,
but we have limited resources and must make some choices.
>> I also notice that you submitted 7 patches in this CF but are
>> reviewing zero.
>
> I've volunteered to review a couple of patches at the FOSDEM Developer
> Meeting - I thought Stephen was entering that into the CF app, not sure
> where it got lost.
There are plenty of patches that need review, so go for it.
Regards,
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 05:19 Erik Rijkers <er@xs4all.nl>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2018-03-03 05:19 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On 2018-03-03 01:55, Tomas Vondra wrote:
> Hi there,
>
> attached is an updated patch fixing all the reported issues (a bit more
> about those below).
Hi,
0007-Track-statistics-for-streaming-spilling.patch won't apply. All
the other patches apply ok.
patch complaints with:
patching file doc/src/sgml/monitoring.sgml
patching file src/backend/catalog/system_views.sql
Hunk #1 succeeded at 734 (offset 2 lines).
patching file src/backend/replication/logical/reorderbuffer.c
patching file src/backend/replication/walsender.c
patching file src/include/catalog/pg_proc.h
Hunk #1 FAILED at 2903.
1 out of 1 hunk FAILED -- saving rejects to file
src/include/catalog/pg_proc.h.rej
patching file src/include/replication/reorderbuffer.h
patching file src/include/replication/walsender_private.h
patching file src/test/regress/expected/rules.out
Hunk #1 succeeded at 1861 (offset 2 lines).
Attached the produced reject file.
thanks,
Erik Rijkers
Attachments:
pg_proc.h.rej (text/x-diff, 1.9K) download
--- src/include/catalog/pg_proc.h
+++ src/include/catalog/pg_proc.h
@@ -2903,7 +2903,7 @@
DESCR("statistics: information about currently active backends");
DATA(insert OID = 3318 ( pg_stat_get_progress_info PGNSP PGUID 12 1 100 0 0 f f f f t t s r 1 0 2249 "25" "{25,23,26,26,20,20,20,20,20,20,20,20,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}" _null_ _null_ pg_stat_get_progress_info _null_ _null_ _null_ ));
DESCR("statistics: information about progress of backends running maintenance command");
-DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25}" "{o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
+DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
DESCR("statistics: information about currently active replication");
DATA(insert OID = 3317 ( pg_stat_get_wal_receiver PGNSP PGUID 12 1 0 0 0 f f f f f f s r 0 0 2249 "" "{23,25,3220,23,3220,23,1184,1184,3220,1184,25,25}" "{o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,status,receive_start_lsn,receive_start_tli,received_lsn,received_tli,last_msg_send_time,last_msg_receipt_time,latest_end_lsn,latest_end_time,slot_name,conninfo}" _null_ _null_ pg_stat_get_wal_receiver _null_ _null_ _null_ ));
DESCR("statistics: information about WAL receiver");
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-03 14:52 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Erik Rijkers <er@xs4all.nl>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-03-03 14:52 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On 03/03/2018 06:19 AM, Erik Rijkers wrote:
> On 2018-03-03 01:55, Tomas Vondra wrote:
>> Hi there,
>>
>> attached is an updated patch fixing all the reported issues (a bit more
>> about those below).
>
> Hi,
>
> 0007-Track-statistics-for-streaming-spilling.patch won't apply. All
> the other patches apply ok.
>
> patch complaints with:
>
> patching file doc/src/sgml/monitoring.sgml
> patching file src/backend/catalog/system_views.sql
> Hunk #1 succeeded at 734 (offset 2 lines).
> patching file src/backend/replication/logical/reorderbuffer.c
> patching file src/backend/replication/walsender.c
> patching file src/include/catalog/pg_proc.h
> Hunk #1 FAILED at 2903.
> 1 out of 1 hunk FAILED -- saving rejects to file
> src/include/catalog/pg_proc.h.rej
> patching file src/include/replication/reorderbuffer.h
> patching file src/include/replication/walsender_private.h
> patching file src/test/regress/expected/rules.out
> Hunk #1 succeeded at 1861 (offset 2 lines).
>
> Attached the produced reject file.
>
Yeah, that's due to fd1a421fe66 which changed columns in pg_proc.h.
Attached is a rebased patch, fixing this.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer.patch.gz (application/gzip, 9.9K) download
0002-Immediatel-WAL-log-assignments.patch.gz (application/gzip, 6.7K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 10.8K) download
0006-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 19.6K) download
0007-Track-statistics-for-streaming-spilling.patch.gz (application/gzip, 4.3K) download
0008-BUGFIX-make-sure-subxact-is-marked-as-is_known_as_su.patch.gz (application/gzip, 570B) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch.gz (application/gzip, 626B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-09 16:07 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-03-09 16:07 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
I think this patch is not going to be ready for PG11.
- It depends on some work in the thread "logical decoding of two-phase
transactions", which is still in progress.
- Various details in the logical_work_mem patch (0001) are unresolved.
- This being partially a performance feature, we haven't seen any
performance tests (e.g., which settings result in which latencies under
which workloads).
That said, the feature seems useful and desirable, and the
implementation makes sense. There are documentation and tests. But
there is a significant amount of design and coding work still necessary.
Attached is a fixup patch that I needed to make it compile.
The last two patches in your series (0008, 0009) are labeled as bug
fixes. Would you like to argue that they should be applied
independently of the rest of the feature?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From 7ac3c2b16f9976c75a0feea3131a36bdf50da2f8 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Fri, 9 Mar 2018 10:50:33 -0500
Subject: [PATCH] fixup! Track statistics for streaming/spilling
---
src/include/catalog/pg_proc.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 9d6c88f0c1..f1cea24379 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2901,7 +2901,7 @@ DATA(insert OID = 2022 ( pg_stat_get_activity PGNSP PGUID 12 1 100 0 0 f f f
DESCR("statistics: information about currently active backends");
DATA(insert OID = 3318 ( pg_stat_get_progress_info PGNSP PGUID 12 1 100 0 0 f f f t t s r 1 0 2249 "25" "{25,23,26,26,20,20,20,20,20,20,20,20,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}" _null_ _null_ pg_stat_get_progress_info _null_ _null_ _null_ ));
DESCR("statistics: information about progress of backends running maintenance command");
-DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
+DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
DESCR("statistics: information about currently active replication");
DATA(insert OID = 3317 ( pg_stat_get_wal_receiver PGNSP PGUID 12 1 0 0 0 f f f f f s r 0 0 2249 "" "{23,25,3220,23,3220,23,1184,1184,3220,1184,25,25}" "{o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,status,receive_start_lsn,receive_start_tli,received_lsn,received_tli,last_msg_send_time,last_msg_receipt_time,latest_end_lsn,latest_end_time,slot_name,conninfo}" _null_ _null_ pg_stat_get_wal_receiver _null_ _null_ _null_ ));
DESCR("statistics: information about WAL receiver");
--
2.16.2
Attachments:
0001-fixup-Track-statistics-for-streaming-spilling.patch (text/plain, 2.5K) download
From 7ac3c2b16f9976c75a0feea3131a36bdf50da2f8 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Fri, 9 Mar 2018 10:50:33 -0500
Subject: [PATCH] fixup! Track statistics for streaming/spilling
---
src/include/catalog/pg_proc.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 9d6c88f0c1..f1cea24379 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2901,7 +2901,7 @@ DATA(insert OID = 2022 ( pg_stat_get_activity PGNSP PGUID 12 1 100 0 0 f f f
DESCR("statistics: information about currently active backends");
DATA(insert OID = 3318 ( pg_stat_get_progress_info PGNSP PGUID 12 1 100 0 0 f f f t t s r 1 0 2249 "25" "{25,23,26,26,20,20,20,20,20,20,20,20,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{cmdtype,pid,datid,relid,param1,param2,param3,param4,param5,param6,param7,param8,param9,param10}" _null_ _null_ pg_stat_get_progress_info _null_ _null_ _null_ ));
DESCR("statistics: information about progress of backends running maintenance command");
-DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
+DATA(insert OID = 3099 ( pg_stat_get_wal_senders PGNSP PGUID 12 1 10 0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null_ ));
DESCR("statistics: information about currently active replication");
DATA(insert OID = 3317 ( pg_stat_get_wal_receiver PGNSP PGUID 12 1 0 0 0 f f f f f s r 0 0 2249 "" "{23,25,3220,23,3220,23,1184,1184,3220,1184,25,25}" "{o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,status,receive_start_lsn,receive_start_tli,received_lsn,received_tli,last_msg_send_time,last_msg_receipt_time,latest_end_lsn,latest_end_time,slot_name,conninfo}" _null_ _null_ pg_stat_get_wal_receiver _null_ _null_ _null_ ));
DESCR("statistics: information about WAL receiver");
--
2.16.2
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-03-29 16:34 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2 siblings, 0 replies; 536+ messages in thread
From: Konstantin Knizhnik @ 2018-03-29 16:34 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 11.01.2018 22:41, Peter Eisentraut wrote:
> On 12/22/17 23:57, Tomas Vondra wrote:
>> PART 1: adding logical_work_mem memory limit (0001)
>> ---------------------------------------------------
>>
>> Currently, limiting the amount of memory consumed by logical decoding is
>> tricky (or you might say impossible) for several reasons:
> I would like to see some more discussion on this, but I think not a lot
> of people understand the details, so I'll try to write up an explanation
> here. This code is also somewhat new to me, so please correct me if
> there are inaccuracies, while keeping in mind that I'm trying to simplify.
>
> The data in the WAL is written as it happens, so the changes belonging
> to different transactions are all mixed together. One of the jobs of
> logical decoding is to reassemble the changes belonging to each
> transaction. The top-level data structure for that is the infamous
> ReorderBuffer. So as it reads the WAL and sees something about a
> transaction, it keeps a copy of that change in memory, indexed by
> transaction ID (ReorderBufferChange). When the transaction commits, the
> accumulated changes are passed to the output plugin and then freed. If
> the transaction aborts, then changes are just thrown away.
>
> So when logical decoding is active, a copy of the changes for each
> active transaction is kept in memory (once per walsender).
>
> More precisely, the above happens for each subtransaction. When the
> top-level transaction commits, it finds all its subtransactions in the
> ReorderBuffer, reassembles everything in the right order, then invokes
> the output plugin.
>
> All this could end up using an unbounded amount of memory, so there is a
> mechanism to spill changes to disk. The way this currently works is
> hardcoded, and this patch proposes to change that.
>
> Currently, when a transaction or subtransaction has accumulated 4096
> changes, it is spilled to disk. When the top-level transaction commits,
> things are read back from disk to do the final processing mentioned above.
>
> This all works mostly fine, but you can construct some more extreme
> cases where this can blow up.
>
> Here is a mundane example. Let's say a change entry takes 100 bytes (it
> might contain a new row, or an update key and some new column values,
> for example). If you have 100 concurrent active sessions and no
> subtransactions, then logical decoding memory is bounded by 4096 * 100 *
> 100 = 40 MB (per walsender) before things spill to disk.
>
> Now let's say you are using a lot of subtransactions, because you are
> using PL functions, exception handling, triggers, doing batch updates.
> If you have 200 subtransactions on average per concurrent session, the
> memory usage bound in that case would be 4096 * 100 * 100 * 200 = 8 GB
> (per walsender). And so on. If you have more concurrent sessions or
> larger changes or more subtransactions, you'll use much more than those
> 8 GB. And if you don't have those 8 GB, then you're stuck at this point.
>
> That is the consideration when we record changes, but we also need
> memory when we do the final processing at commit time. That is slightly
> less problematic because we only process one top-level transaction at a
> time, so the formula is only 4096 * avg_size_of_changes * nr_subxacts
> (without the concurrent sessions factor).
>
> So, this patch proposes to improve this as follows:
>
> - We compute the actual size of each ReorderBufferChange and keep a
> running tally for each transaction, instead of just counting the number
> of changes.
>
> - We have a configuration setting that allows us to change the limit
> instead of the hardcoded 4096. The configuration setting is also in
> terms of memory, not in number of changes.
>
> - The configuration setting is for the total memory usage per decoding
> session, not per subtransaction. (So we also keep a running tally for
> the entire ReorderBuffer.)
>
> There are two open issues with this patch:
>
> One, this mechanism only applies when recording changes. The processing
> at commit time still uses the previous hardcoded mechanism. The reason
> for this is, AFAIU, that as things currently work, you have to have all
> subtransactions in memory to do the final processing. There are some
> proposals to change this as well, but they are more involved. Arguably,
> per my explanation above, memory use at commit time is less likely to be
> a problem.
>
> Two, what to do when the memory limit is reached. With the old
> accounting, this was easy, because we'd decide for each subtransaction
> independently whether to spill it to disk, when it has reached its 4096
> limit. Now, we are looking at a global limit, so we have to find a
> transaction to spill in some other way. The proposed patch searches
> through the entire list of transactions to find the largest one. But as
> the patch says:
>
> "XXX With many subtransactions this might be quite slow, because we'll
> have to walk through all of them. There are some options how we could
> improve that: (a) maintain some secondary structure with transactions
> sorted by amount of changes, (b) not looking for the entirely largest
> transaction, but e.g. for transaction using at least some fraction of
> the memory limit, and (c) evicting multiple transactions at once, e.g.
> to free a given portion of the memory limit (e.g. 50%)."
>
> (a) would create more overhead for the case where everything fits into
> memory, so it seems unattractive. Some combination of (b) and (c) seems
> useful, but we'd have to come up with something concrete.
>
> Thoughts?
>
I am very sorry that I have not noticed this thread before.
Spilling to the file in reorder buffer is the main factor limiting speed
of importing data in multimaster and shardman (sharding based on FDW
with redundancy provided by LR).
This is why we think a lot about possible ways of addressing this issue.
Right now data of huge transaction is written to the disk three times
before it is applied at replica. And obviously read also three times.
First it is saved in WAL, then spilled to the disk by reorder buffer and
once again spilled to the disk at replica before assignment to the
particular apply worker
(last one is specific of multimaster, which can apply received
transactions concurrently).
We considered three different approaches:
1. Streaming. It is similar with the proposed patch, the main difference
is that we do not want to spill transaction in temporary file at
replica, but apply it immediately in separate backend and abort
transaction if it is aborted at master. Certainly it will work only with
2PC.
2. Elimination of spilling by rescanning WAL.
3. Bypass WAL: add hooks to heapam to buffer and propagate changes
immediately to replica and apply them in dedicated backend.
I have implemented prototype of such replication. With one replica it
shows about 1.5x slowdown comparing with standalone/async LR and about
2-3 improvement comparing with sync LR. For two replicas result is 2x
slower than async LR and 2-8 times faster than sync LR (depending on
number of concurrent connections).
Approach 3) seems to be specific to multimaster/shardman, so most likely
it can not be considered for general LR.
So I want to compare 1 and 2. Did you ever though about something like 2?
Right now in the proposed patch you just move spilling to the file from
master to replica.
It still can make sense to avoid memory overflow and reduce disk IO at
master.
But if we have just one huge transaction (COPY) importing gigabytes of
data to the database,
then performance will be almost the same with your patch or without it.
The only difference is where we serialize transaction: at master or at
replica side.
In this sense this patch doesn't solve the problem with slow load of
large bulks of data though LR.
Alternatively (approach 2) we can have small in-memory buffer for
decoding transaction and remember LSN and snapshot of this transaction
start.
In case of buffer overflow we just continue WAL traversal until we reach
end of the transaction. After it we restart scanning WAL from the
beginning of this transaction at this second pass
send changes directly to the output plugin. So we have to scan WAL
several times but do not need to spill anything to the disk neither at
publisher, neither at subscriber side.
Certainly this approach will be inefficient if we have several long
interleaving transactions. But in most customer's use cases we have
observed until now there is just one huge transaction performing bulk load.
May be I missed something, but this approach seems to be easier for
implementation than transaction streaming. And it doesn't require any
changes in output plugin API.
I realize that it is a little bit late to ask this question once your
patch is almost ready, but what do you think about it? Are there some
pitfals with this approach?
There is one more aspect and performance problem with LR we have faced
with shardman: if there are several publications for different subsets
of table at one instance,
then WAL senders have to do a lot of useless work. Them are decoding
transactions which have no relation to this publication. But WAL sender
doesn't know it until it reaches the end of this transaction. What is
worser: if transaction is huge, then all WAL senders will spill it to
the disk even through only one of them actually needs it. So data of
huge transaction is written not three times, but N times, where N is
number of publications. The only solution of the problem we can imagine
is to let backend somehow inform WAL sender (through shared message queue?)
about LSN-s it should considered. In this case WAL sender can skip large
portions of WAL without decoding. We also want to know opinion of
2ndQuandarnt about this idea.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-07-01 11:43 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-07-01 11:43 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
This patch set was not updated for the 2018-07 commitfest, so moved to -09.
On 09.03.18 17:07, Peter Eisentraut wrote:
> I think this patch is not going to be ready for PG11.
>
> - It depends on some work in the thread "logical decoding of two-phase
> transactions", which is still in progress.
>
> - Various details in the logical_work_mem patch (0001) are unresolved.
>
> - This being partially a performance feature, we haven't seen any
> performance tests (e.g., which settings result in which latencies under
> which workloads).
>
> That said, the feature seems useful and desirable, and the
> implementation makes sense. There are documentation and tests. But
> there is a significant amount of design and coding work still necessary.
>
> Attached is a fixup patch that I needed to make it compile.
>
> The last two patches in your series (0008, 0009) are labeled as bug
> fixes. Would you like to argue that they should be applied
> independently of the rest of the feature?
>
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-10-02 04:59 Michael Paquier <michael@paquier.xyz>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Michael Paquier @ 2018-10-02 04:59 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On Sat, Mar 03, 2018 at 03:52:40PM +0100, Tomas Vondra wrote:
> Yeah, that's due to fd1a421fe66 which changed columns in pg_proc.h.
> Attached is a rebased patch, fixing this.
The latest patch set does not apply anymore, and had no activity for the
last two months, so I am marking it as returned with feedback.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-16 14:31 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-12-16 14:31 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
Hi,
Attached is an updated version of this patch series. It's meant to be
applied on top of the 2pc decoding patch [1], because streaming of
in-progress transactions requires handling of concurrent aborts. So it
may or may not apply directly to master, I'm not sure - unfortunately
that's likely to confuse the cputube thing, but I don't want to include
the 2pc decoding bits here because that would be just confusing.
If needed, the part introducing logical_work_mem limit for ReorderBuffer
can be separated and committed independently, but I do expect this to be
committed after the 2pc decoding patch so I've left it like this.
This new version is mostly just a rebase to current master (or almost,
because 2pc decoding only applies to 29180e5d78 due to minor bitrot),
but it also addresses the new stuff committed since last version (most
importantly decoding of TRUNCATE). It also fixes a bug in WAL-logging of
subxact assignments, where the assignment was included in records with
XID=0, essentially failing to track the subxact properly.
For the logical_work_mem part, I think this is quite solid. The main
question is how to pick transactions for eviction. For now it uses the
same approach as master (i.e. picking the largest top-level transaction,
although measured by amount of memory and not just number of changes).
But I've realized that may not work with Generation context that great,
because unlike AllocSet it does not reuse the memory. That's nice as it
allows freeing old blocks (which AllocSet can't), but it means a small
transaction can have a change on old blocks preventing free(). That is
something we have in pg11 already, because that's where Generation
context got introduced - I haven't seen this issue in practice, but we
might need to do something about it.
In any case, I'm thinking we may need to pick a different eviction
algorithm - say using a transaction with the oldest change (and loop
until we release at least one block in the Generation context), or maybe
look for block mixing changes from the smallest number of transactions,
or something like that. Other ideas are welcome. I don't think the exact
algorithm is particularly critical, because it's meant to be triggered
only very rarely (i.e. pick logical_work_mem high enough).
The in-progress streaming is mostly mechanical extension of existing
functionality (new methods in various APIs, ...) and refactoring of
ReorderBuffer to handle incremental decoding. I'm sure it'd benefit from
reviews, of course.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_work_mem-to-limit-ReorderBuffer-20181216.patch.gz (application/gzip, 9.9K) download
0002-Immediately-WAL-log-assignments-20181216.patch.gz (application/gzip, 6.3K) download
0003-Issue-individual-invalidations-with-wal_lev-20181216.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-me-20181216.patch.gz (application/gzip, 5.7K) download
0005-Implement-streaming-mode-in-ReorderBuffer-20181216.patch.gz (application/gzip, 11.2K) download
0006-Add-support-for-streaming-to-built-in-repli-20181216.patch.gz (application/gzip, 19.7K) download
0007-Track-statistics-for-streaming-spilling-20181216.patch.gz (application/gzip, 4.2K) download
0008-BUGFIX-set-final_lsn-for-subxacts-before-cl-20181216.patch.gz (application/gzip, 637B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-16 17:54 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2018-12-16 17:54 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
FWIW the original CF entry in 2018-07 [1] was marked as RWF. I'm not
sure what's the right way to resubmit such patches, so I've created a
new entry in 2019-01 [2] referencing the same hackers thread (and with
the same authors/reviewers metadata).
[1] https://commitfest.postgresql.org/19/1429/
[2] https://commitfest.postgresql.org/21/1927/
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-17 16:23 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2018-12-17 16:23 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Tomas,
> This new version is mostly just a rebase to current master (or almost,
> because 2pc decoding only applies to 29180e5d78 due to minor bitrot),
> but it also addresses the new stuff committed since last version (most
> importantly decoding of TRUNCATE). It also fixes a bug in WAL-logging of
> subxact assignments, where the assignment was included in records with
> XID=0, essentially failing to track the subxact properly.
I started reviewing your patch about a month ago and tried to do an
in-depth review, since I am very interested in this patch too. The new
version is not applicable to master 29180e5d78, but everything is OK
after applying 2pc patch before. Anyway, I guess it may complicate
further testing and review, since any potential reviewer has to take
into account both patches at once. Previous version was applicable to
master and was working fine for me separately (excepting a few
patch-specific issues, which I try to explain below).
Patch review
========
First of all, I want to say thank you for such a huge work done. Here
are some problems, which I have found and hopefully fixed with my
additional patch (please, find attached, it should be applicable to the
last commit of your newest patch version):
1) The most important issue is that your tap tests were broken—there was
missing option "WITH (streaming=true)" in the subscription creating
statement. Therefore, spilling mechanism has been tested rather than
streaming.
2) After fixing tests the first one with simple streaming is immediately
failed, because of logical replication worker segmentation fault. It
happens, since worker tries to call stream_cleanup_files inside
stream_open_file at the stream start, while nxids is zero, then it goes
to the negative value and everything crashes. Something similar may
happen with xids array, so I added two checks there.
3) The next problem is much more critical and is dedicated to historic
MVCC visibility rules. Previously, walsender was starting to decode
transaction on commit and we were able to resolve all xmin, xmax,
combocids to cmin/cmax, build tuplecids hash and so on, but now we start
doing all these things on the fly.
Thus, rather difficult situation arises: HeapTupleSatisfiesHistoricMVCC
is trying to validate catalog tuples, which are currently in the future
relatively to the current decoder position inside transaction, e.g. we
may want to resolve cmin/cmax of a tuple, which was created with cid 3
and deleted with cid 5, while we are currently at cid 4, so our
tuplecids hash is not complete to handle such a case.
I have updated HeapTupleSatisfiesHistoricMVCC visibility rules with two
options:
/*
* If we accidentally see a tuple from our transaction, but cannot
resolve its
* cmin, so probably it is from the future, thus drop it.
*/
if (!resolved)
return false;
and
/*
* If we accidentally see a tuple from our transaction, but cannot
resolve its
* cmax or cmax == InvalidCommandId, so probably it is still valid,
thus accept it.
*/
if (!resolved || cmax == InvalidCommandId)
return true;
4) There was a problem with marking top-level transaction as having
catalog changes if one of its subtransactions has. It was causing a
problem with DDL statements just after subtransaction start (savepoint),
so data from new columns is not replicated.
5) Similar issue with schema send. You send schema only once per each
sub/transaction (IIRC), while we have to update schema on each catalog
change: invalidation execution, snapshot rebuild, adding new tuple cids.
So I ended up with adding is_schema_send flag to ReorderBufferTXN, since
it is easy to set it inside RB and read in the output plugin. Probably,
we have to choose a better place for this flag.
6) To better handle all these tricky cases I added new tap
test—014_stream_tough_ddl.pl—which consist of really tough combination
of DDL, DML, savepoints and ROLLBACK/RELEASE in a single transaction.
I marked all my fixes and every questionable place with comment and
"TOCHECK:" label for easy search. Removing of pretty any of these fixes
leads to the tests fail due to the segmentation fault or replication
mismatch. Though I mostly read and tested old version of patch, but
after a quick look it seems that all these fixes are applicable to the
new version of patch as well.
Performance
========
I have also performed a series of performance tests, and found that
patch adds a huge overhead in the case of a large transaction consisting
of many small rows, e.g.:
CREATE TABLE large_test (num1 bigint, num2 double precision, num3 double
precision);
EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1, num2, num3)
SELECT round(random()*10), random(), random()*142
FROM generate_series(1, 1000000) s(i);
Execution Time: 2407.709 ms
Total Time: 11494,238 ms (00:11,494)
With synchronous_standby_names and 64 MB logical_work_mem it takes up to
x5 longer, while without patch it is about x2. Thus, logical replication
streaming is approximately x4 as slower for similar transactions.
However, dealing with large transactions consisting of a small number of
large rows is much better:
CREATE TABLE large_text (t TEXT);
EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_text
SELECT (SELECT string_agg('x', ',')
FROM generate_series(1, 1000000)) FROM generate_series(1, 125);
Execution Time: 3545.642 ms
Total Time: 7678,617 ms (00:07,679)
It is around the same x2 as without patch. If someone is interested I
also added flamegraphs of walsender and logical replication worker
during first numerical transaction processing.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
logical_repl_worker_new_perf.svg.zip (application/zip, 56.0K) download
walsender_new_perf.svg.zip (application/zip, 34.7K) download
0009-Fix-worker-historic-MVCC-visibility-rules-subxacts-s.patch (text/x-patch, 20.4K) download
From 0a7cbaeaa179bde9dd5146e60f7b5c52b6c34899 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <alex.lumir@gmail.com>
Date: Mon, 17 Dec 2018 15:43:13 +0300
Subject: [PATCH] Fix worker, historic MVCC visibility rules, subxacts, schema
send, tests
---
doc/src/sgml/logicaldecoding.sgml | 2 +-
.../replication/logical/reorderbuffer.c | 61 +++++++++---
src/backend/replication/logical/worker.c | 25 ++---
src/backend/replication/pgoutput/pgoutput.c | 20 +++-
src/backend/replication/walsender.c | 2 +-
src/backend/utils/time/tqual.c | 16 ++-
src/include/replication/reorderbuffer.h | 5 +
..._stream_simple.pl => 011_stream_simple.pl} | 2 +-
...tream_subxact.pl => 012_stream_subxact.pl} | 2 +-
.../{011_stream_ddl.pl => 013_stream_ddl.pl} | 2 +-
.../subscription/t/014_stream_tough_ddl.pl | 98 +++++++++++++++++++
...t_abort.pl => 015_stream_subxact_abort.pl} | 2 +-
...ort.pl => 016_stream_subxact_ddl_abort.pl} | 2 +-
13 files changed, 200 insertions(+), 39 deletions(-)
rename src/test/subscription/t/{009_stream_simple.pl => 011_stream_simple.pl} (98%)
rename src/test/subscription/t/{010_stream_subxact.pl => 012_stream_subxact.pl} (98%)
rename src/test/subscription/t/{011_stream_ddl.pl => 013_stream_ddl.pl} (98%)
create mode 100644 src/test/subscription/t/014_stream_tough_ddl.pl
rename src/test/subscription/t/{012_stream_subxact_abort.pl => 015_stream_subxact_abort.pl} (97%)
rename src/test/subscription/t/{013_stream_subxact_ddl_abort.pl => 016_stream_subxact_ddl_abort.pl} (97%)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 3571f96a8d..dbec2e4ef7 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -1057,7 +1057,7 @@ stream_commit_cb(...); <-- commit of the streamed transaction
</para>
<para>
- Similarly to spill-to-disk behavior, sStreaming is triggered when the total
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
amount of changes decoded from the WAL (for all in-progress transactions)
exceeds limit defined by <varname>logical_work_mem</varname> setting. At
that point the largest toplevel transaction (measured by amount of memory
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1c394296ac..2200332999 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1744,9 +1744,10 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
else
{
+ // TOCHECK: Is the second assert actually necessary?
Assert(ent->cmin == change->data.tuplecid.cmin);
- Assert(ent->cmax == InvalidCommandId ||
- ent->cmax == change->data.tuplecid.cmax);
+ // Assert(ent->cmax == InvalidCommandId ||
+ // ent->cmax == change->data.tuplecid.cmax);
/*
* if the tuple got valid in this transaction and now got deleted
@@ -2881,6 +2882,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2895,6 +2899,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -3476,7 +3497,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* using snapshot half-way through the subxact.
*/
command_id = txn->command_id;
- snapshot_now = txn->snapshot_now;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ // snapshot_now = txn->snapshot_now;
}
/*
@@ -3522,7 +3551,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Enforce correct ordering of changes, merged from multiple
* subtransactions. The changes may have the same LSN due to
- * MULTI_INSERT xllog records.
+ * MULTI_INSERT xlog records.
*/
if (prev_lsn != InvalidXLogRecPtr)
Assert(prev_lsn <= change->lsn);
@@ -3731,6 +3760,11 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
snapshot_now = change->data.snapshot;
}
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
/* and continue with the new one */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
@@ -3868,7 +3902,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->streamTxns += (rbtxn_is_streamed(txn)) ? 1 : 0;
rb->streamBytes += txn->size;
- elog(WARNING, "updating stream stats %p %ld %ld %ld",
+ elog(INFO, "updating stream stats %p %ld %ld %ld",
rb, rb->streamCount, rb->streamTxns, txn->size);
/*
@@ -4950,7 +4984,7 @@ ResolveCminCmaxDuringDecoding(HTAB *tuplecid_data,
CommandId *cmin, CommandId *cmax)
{
ReorderBufferTupleCidKey key;
- ReorderBufferTupleCidEnt *ent;
+ ReorderBufferTupleCidEnt *ent = NULL;
ForkNumber forkno;
BlockNumber blockno;
bool updated_mapping = false;
@@ -4974,11 +5008,16 @@ ResolveCminCmaxDuringDecoding(HTAB *tuplecid_data,
&key.tid);
restart:
- ent = (ReorderBufferTupleCidEnt *)
- hash_search(tuplecid_data,
- (void *) &key,
- HASH_FIND,
- NULL);
+ /*
+ * TOCHECK: If tuplecid_data is NULL, then we are not able to resolve cmin/cmax,
+ * so try to update mappings and return false.
+ */
+ if (tuplecid_data != NULL)
+ ent = (ReorderBufferTupleCidEnt *)
+ hash_search(tuplecid_data,
+ (void *) &key,
+ HASH_FIND,
+ NULL);
/*
* failed to find a mapping, check whether the table was rewritten and
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index eaefba2049..adf69a5a38 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2444,10 +2444,21 @@ stream_open_file(Oid subid, TransactionId xid, bool first_segment)
{
MemoryContext oldcxt;
- stream_cleanup_files(subid, xid);
+ /*
+ * TOCHECK: If nxids=0, then we have nothing to clean up.
+ */
+ if (nxids > 0)
+ stream_cleanup_files(subid, xid);
oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+ /* TOCHECK: Initialize xids array if it is the first run. */
+ if (xids == NULL)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+
/*
* We need to remember the XIDs we spilled to files, so that we can
* remove them at worker exit (e.g. after DROP SUBSCRIPTION).
@@ -2462,16 +2473,8 @@ stream_open_file(Oid subid, TransactionId xid, bool first_segment)
*/
if (nxids == maxnxids) /* array of XIDs is full */
{
- if (!xids)
- {
- maxnxids = 64;
- xids = palloc(maxnxids * sizeof(TransactionId));
- }
- else
- {
- maxnxids = 2 * maxnxids;
- xids = repalloc(xids, maxnxids * sizeof(TransactionId));
- }
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
}
xids[nxids++] = xid;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 04432ffb57..d5a3cf6308 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -370,7 +370,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
static void
maybe_send_schema(LogicalDecodingContext *ctx,
TransactionId topxid, TransactionId xid,
- Relation relation, RelationSyncEntry *relentry)
+ Relation relation, RelationSyncEntry *relentry, ReorderBufferTXN *txn)
{
bool schema_sent = relentry->schema_sent;
@@ -381,7 +381,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
* that we don't know at this point.
*/
if (in_streaming)
- schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ // schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
if (!schema_sent)
{
@@ -415,7 +423,9 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->xid = xid;
if (in_streaming)
- set_schema_sent_in_streamed_txn(relentry, topxid);
+ /* TOCHECK: Maybe change flag location? */
+ txn->is_schema_sent = true;
+ // set_schema_sent_in_streamed_txn(relentry, topxid);
else
relentry->schema_sent = true;
}
@@ -479,7 +489,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, topxid, xid, relation, relentry);
+ maybe_send_schema(ctx, topxid, xid, relation, relentry, txn);
/* Send the data */
switch (change->action)
@@ -569,7 +579,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, topxid, xid, relation, relentry);
+ maybe_send_schema(ctx, topxid, xid, relation, relentry, txn);
}
if (nrelids > 0)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index be92f3e5f5..de709250c0 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -3621,7 +3621,7 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->streamCount = rb->streamCount;
MyWalSnd->streamBytes = rb->streamBytes;
- elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ elog(INFO, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
rb->streamTxns, rb->streamCount, rb->streamBytes);
diff --git a/src/backend/utils/time/tqual.c b/src/backend/utils/time/tqual.c
index f7c4c9188c..b905077164 100644
--- a/src/backend/utils/time/tqual.c
+++ b/src/backend/utils/time/tqual.c
@@ -1692,8 +1692,12 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * TOCHECK: If we accidentally see a tuple from our transaction, but cannot resolve its
+ * cmin, so probably it is from the future, thus drop it.
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1763,10 +1767,12 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * TOCHECK: If we accidentally see a tuple from our transaction, but cannot resolve its
+ * cmax or cmax == InvalidCommandId, so probably it is still valid, thus accept it.
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index bf5766c1e4..29dba6673c 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -220,6 +220,11 @@ typedef struct ReorderBufferTXN
/* In case of 2PC we need to pass GID to output plugin */
char *gid;
+ /*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
/*
* Toplevel transaction for this subxact (NULL for top-level).
*/
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/011_stream_simple.pl
similarity index 98%
rename from src/test/subscription/t/009_stream_simple.pl
rename to src/test/subscription/t/011_stream_simple.pl
index 4d01f7e5ec..f0aae1041a 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/011_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/012_stream_subxact.pl
similarity index 98%
rename from src/test/subscription/t/010_stream_subxact.pl
rename to src/test/subscription/t/012_stream_subxact.pl
index 1a8b8ffe9e..00dd60b91f 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/012_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/013_stream_ddl.pl
similarity index 98%
rename from src/test/subscription/t/011_stream_ddl.pl
rename to src/test/subscription/t/013_stream_ddl.pl
index 04af0900ac..ecaf4383b1 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/013_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/014_stream_tough_ddl.pl b/src/test/subscription/t/014_stream_tough_ddl.pl
new file mode 100644
index 0000000000..02969c7260
--- /dev/null
+++ b/src/test/subscription/t/014_stream_tough_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/015_stream_subxact_abort.pl
similarity index 97%
rename from src/test/subscription/t/012_stream_subxact_abort.pl
rename to src/test/subscription/t/015_stream_subxact_abort.pl
index 6fecfe6fe7..dcec081fa1 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/015_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/016_stream_subxact_ddl_abort.pl
similarity index 97%
rename from src/test/subscription/t/013_stream_subxact_ddl_abort.pl
rename to src/test/subscription/t/016_stream_subxact_ddl_abort.pl
index 50990c170c..41ad2b668c 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/016_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
);
wait_for_caught_up($node_publisher, $appname);
--
2.17.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-17 22:28 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-12-17 22:28 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Alexey,
Thanks for the thorough and extremely valuable review!
On 12/17/18 5:23 PM, Alexey Kondratov wrote:
> Hi Tomas,
>
>> This new version is mostly just a rebase to current master (or almost,
>> because 2pc decoding only applies to 29180e5d78 due to minor bitrot),
>> but it also addresses the new stuff committed since last version (most
>> importantly decoding of TRUNCATE). It also fixes a bug in WAL-logging of
>> subxact assignments, where the assignment was included in records with
>> XID=0, essentially failing to track the subxact properly.
>
> I started reviewing your patch about a month ago and tried to do an
> in-depth review, since I am very interested in this patch too. The new
> version is not applicable to master 29180e5d78, but everything is OK
> after applying 2pc patch before. Anyway, I guess it may complicate
> further testing and review, since any potential reviewer has to take
> into account both patches at once. Previous version was applicable to
> master and was working fine for me separately (excepting a few
> patch-specific issues, which I try to explain below).
>
I agree it's somewhat annoying, but I don't think there's a better way,
unfortunately. Decoding in-progress transactions does require safe
handling of concurrent aborts, so it has to be committed after the 2pc
decoding patch (which makes that possible). But the 2pc patch also
touches the same places as this patch series (it reworks the reorder
buffer for example).
>
> Patch review
> ========
>
> First of all, I want to say thank you for such a huge work done. Here
> are some problems, which I have found and hopefully fixed with my
> additional patch (please, find attached, it should be applicable to the
> last commit of your newest patch version):
>
> 1) The most important issue is that your tap tests were broken—there was
> missing option "WITH (streaming=true)" in the subscription creating
> statement. Therefore, spilling mechanism has been tested rather than
> streaming.
>
D'oh!
> 2) After fixing tests the first one with simple streaming is immediately
> failed, because of logical replication worker segmentation fault. It
> happens, since worker tries to call stream_cleanup_files inside
> stream_open_file at the stream start, while nxids is zero, then it goes
> to the negative value and everything crashes. Something similar may
> happen with xids array, so I added two checks there.
>
> 3) The next problem is much more critical and is dedicated to historic
> MVCC visibility rules. Previously, walsender was starting to decode
> transaction on commit and we were able to resolve all xmin, xmax,
> combocids to cmin/cmax, build tuplecids hash and so on, but now we start
> doing all these things on the fly.
>
> Thus, rather difficult situation arises: HeapTupleSatisfiesHistoricMVCC
> is trying to validate catalog tuples, which are currently in the future
> relatively to the current decoder position inside transaction, e.g. we
> may want to resolve cmin/cmax of a tuple, which was created with cid 3
> and deleted with cid 5, while we are currently at cid 4, so our
> tuplecids hash is not complete to handle such a case.
>
Damn it! I ran into those two issues some time ago and I fixed it, but
I've forgotten to merge that fix into the patch. I'll merge those fixed
and compare them to your proposed fix, and send a new version tomorrow.
>
> 4) There was a problem with marking top-level transaction as having
> catalog changes if one of its subtransactions has. It was causing a
> problem with DDL statements just after subtransaction start (savepoint),
> so data from new columns is not replicated.
>
> 5) Similar issue with schema send. You send schema only once per each
> sub/transaction (IIRC), while we have to update schema on each catalog
> change: invalidation execution, snapshot rebuild, adding new tuple cids.
> So I ended up with adding is_schema_send flag to ReorderBufferTXN, since
> it is easy to set it inside RB and read in the output plugin. Probably,
> we have to choose a better place for this flag.
>
Hmm. Can you share an example how to trigger these issues?
> 6) To better handle all these tricky cases I added new tap
> test—014_stream_tough_ddl.pl—which consist of really tough combination
> of DDL, DML, savepoints and ROLLBACK/RELEASE in a single transaction.
>
Thanks!
> I marked all my fixes and every questionable place with comment and
> "TOCHECK:" label for easy search. Removing of pretty any of these fixes
> leads to the tests fail due to the segmentation fault or replication
> mismatch. Though I mostly read and tested old version of patch, but
> after a quick look it seems that all these fixes are applicable to the
> new version of patch as well.
>
Thanks. I'll go through your patch tomorrow.
>
> Performance
> ========
>
> I have also performed a series of performance tests, and found that
> patch adds a huge overhead in the case of a large transaction consisting
> of many small rows, e.g.:
>
> CREATE TABLE large_test (num1 bigint, num2 double precision, num3 double
> precision);
>
> EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1, num2, num3)
> SELECT round(random()*10), random(), random()*142
> FROM generate_series(1, 1000000) s(i);
>
> Execution Time: 2407.709 ms
> Total Time: 11494,238 ms (00:11,494)
>
> With synchronous_standby_names and 64 MB logical_work_mem it takes up to
> x5 longer, while without patch it is about x2. Thus, logical replication
> streaming is approximately x4 as slower for similar transactions.
>
> However, dealing with large transactions consisting of a small number of
> large rows is much better:
>
> CREATE TABLE large_text (t TEXT);
>
> EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_text
> SELECT (SELECT string_agg('x', ',')
> FROM generate_series(1, 1000000)) FROM generate_series(1, 125);
>
> Execution Time: 3545.642 ms
> Total Time: 7678,617 ms (00:07,679)
>
> It is around the same x2 as without patch. If someone is interested I
> also added flamegraphs of walsender and logical replication worker
> during first numerical transaction processing.
>
Interesting. Any idea where does the extra overhead in this particular
case come from? It's hard to deduce that from the single flame graph,
when I don't have anything to compare it with (i.e. the flame graph for
the "normal" case).
I'll investigate this (probably not this week), but in general it's good
to keep in mind a couple of things:
1) Some overhead is expected, due to doing things incrementally.
2) The memory limit should be set to sufficiently high value to be hit
only infrequently.
3) And when the limit is actually hit, it's an alternative to spilling
large amounts of data locally (to disk) or incurring significant
replication lag later.
So I'm not particularly worried, but I'll look into that. I'd be much
more worried if there was measurable overhead in cases when there's no
streaming happening (either because it's disabled or the memory limit
was not hit).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-18 14:07 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2018-12-18 14:07 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
On 18.12.2018 1:28, Tomas Vondra wrote:
>> 4) There was a problem with marking top-level transaction as having
>> catalog changes if one of its subtransactions has. It was causing a
>> problem with DDL statements just after subtransaction start (savepoint),
>> so data from new columns is not replicated.
>>
>> 5) Similar issue with schema send. You send schema only once per each
>> sub/transaction (IIRC), while we have to update schema on each catalog
>> change: invalidation execution, snapshot rebuild, adding new tuple cids.
>> So I ended up with adding is_schema_send flag to ReorderBufferTXN, since
>> it is easy to set it inside RB and read in the output plugin. Probably,
>> we have to choose a better place for this flag.
>>
> Hmm. Can you share an example how to trigger these issues?
Test cases inside 014_stream_tough_ddl.pl and old ones (with
streaming=true option added) should reproduce all these issues. In
general, it happens in a txn like:
INSERT
SAVEPOINT
ALTER TABLE ... ADD COLUMN
INSERT
then the second insert may discover old version of catalog.
> Interesting. Any idea where does the extra overhead in this particular
> case come from? It's hard to deduce that from the single flame graph,
> when I don't have anything to compare it with (i.e. the flame graph for
> the "normal" case).
I guess that bottleneck is in disk operations. You can check
logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
writes (~26%) take around 35% of CPU time in summary. To compare,
please, see attached flame graph for the following transaction:
INSERT INTO large_text
SELECT (SELECT string_agg('x', ',')
FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
Execution Time: 44519.816 ms
Time: 98333,642 ms (01:38,334)
where disk IO is only ~7-8% in total. So we get very roughly the same
~x4-5 performance drop here. JFYI, I am using a machine with SSD for tests.
Therefore, probably you may write changes on receiver in bigger chunks,
not each change separately.
> So I'm not particularly worried, but I'll look into that. I'd be much
> more worried if there was measurable overhead in cases when there's no
> streaming happening (either because it's disabled or the memory limit
> was not hit).
What I have also just found, is that if a table row is large enough to
be TOASTed, e.g.:
INSERT INTO large_text
SELECT (SELECT string_agg('x', ',')
FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
then logical_work_mem limit is not hit and we neither stream, nor spill
to disk this transaction, while it is still large. In contrast, the
transaction above (with 1000000 smaller rows) being comparable in size
is streamed. Not sure, that it is easy to add proper accounting of
TOAST-able columns, but it worth it.
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
logical_repl_worker_text_stream_perf.zip (application/zip, 42.1K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-18 23:56 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-12-18 23:56 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Alexey,
Attached is an updated version of the patches, with all the fixes I've
done in the past. I believe it should fix at least some of the issues
you reported - certainly the problem with stream_cleanup_files, but
perhaps some of the other issues too.
I'm a bit confused by the changes to TAP tests. Per the patch summary,
some .pl files get renamed (nor sure why), a new one is added, etc. So
I've instead enabled streaming subscriptions in all tests, which with
this patch produces two failures:
Test Summary Report
-------------------
t/004_sync.pl (Wstat: 7424 Tests: 1 Failed: 0)
Non-zero exit status: 29
Parse errors: Bad plan. You planned 7 tests but ran 1.
t/011_stream_ddl.pl (Wstat: 256 Tests: 2 Failed: 1)
Failed test: 2
Non-zero exit status: 1
So yeah, there's more stuff to fix. But I can't directly apply your
fixes because the updated patches are somewhat different.
On 12/18/18 3:07 PM, Alexey Kondratov wrote:
> On 18.12.2018 1:28, Tomas Vondra wrote:
>>> 4) There was a problem with marking top-level transaction as having
>>> catalog changes if one of its subtransactions has. It was causing a
>>> problem with DDL statements just after subtransaction start (savepoint),
>>> so data from new columns is not replicated.
>>>
>>> 5) Similar issue with schema send. You send schema only once per each
>>> sub/transaction (IIRC), while we have to update schema on each catalog
>>> change: invalidation execution, snapshot rebuild, adding new tuple cids.
>>> So I ended up with adding is_schema_send flag to ReorderBufferTXN, since
>>> it is easy to set it inside RB and read in the output plugin. Probably,
>>> we have to choose a better place for this flag.
>>>
>> Hmm. Can you share an example how to trigger these issues?
>
> Test cases inside 014_stream_tough_ddl.pl and old ones (with
> streaming=true option added) should reproduce all these issues. In
> general, it happens in a txn like:
>
> INSERT
> SAVEPOINT
> ALTER TABLE ... ADD COLUMN
> INSERT
>
> then the second insert may discover old version of catalog.
>
Yeah, that's the issue I've discovered before and thought it got fixed.
>> Interesting. Any idea where does the extra overhead in this particular
>> case come from? It's hard to deduce that from the single flame graph,
>> when I don't have anything to compare it with (i.e. the flame graph for
>> the "normal" case).
>
> I guess that bottleneck is in disk operations. You can check
> logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
> writes (~26%) take around 35% of CPU time in summary. To compare,
> please, see attached flame graph for the following transaction:
>
> INSERT INTO large_text
> SELECT (SELECT string_agg('x', ',')
> FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>
> Execution Time: 44519.816 ms
> Time: 98333,642 ms (01:38,334)
>
> where disk IO is only ~7-8% in total. So we get very roughly the same
> ~x4-5 performance drop here. JFYI, I am using a machine with SSD for tests.
>
> Therefore, probably you may write changes on receiver in bigger chunks,
> not each change separately.
>
Possibly, I/O is certainly a possible culprit, although we should be
using buffered I/O and there certainly are not any fsyncs here. So I'm
not sure why would it be cheaper to do the writes in batches.
BTW does this mean you see the overhead on the apply side? Or are you
running this on a single machine, and it's difficult to decide?
>> So I'm not particularly worried, but I'll look into that. I'd be much
>> more worried if there was measurable overhead in cases when there's no
>> streaming happening (either because it's disabled or the memory limit
>> was not hit).
>
> What I have also just found, is that if a table row is large enough to
> be TOASTed, e.g.:
>
> INSERT INTO large_text
> SELECT (SELECT string_agg('x', ',')
> FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>
> then logical_work_mem limit is not hit and we neither stream, nor spill
> to disk this transaction, while it is still large. In contrast, the
> transaction above (with 1000000 smaller rows) being comparable in size
> is streamed. Not sure, that it is easy to add proper accounting of
> TOAST-able columns, but it worth it.
>
That's certainly strange and possibly a bug in the memory accounting
code. I'm not sure why would that happen, though, because TOAST data
look just like regular INSERT changes. Interesting. I wonder if it's
already fixed in this updated version, but it's a bit too late to
investigate that today.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_work_mem-to-limit-ReorderBuffer-20181219.patch.gz (application/gzip, 9.9K) download
0002-Immediately-WAL-log-assignments-20181219.patch.gz (application/gzip, 6.4K) download
0003-Issue-individual-invalidations-with-wal_lev-20181219.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-me-20181219.patch.gz (application/gzip, 6.1K) download
0005-Implement-streaming-mode-in-ReorderBuffer-20181219.patch.gz (application/gzip, 12.3K) download
0006-Add-support-for-streaming-to-built-in-repli-20181219.patch.gz (application/gzip, 20.1K) download
0007-Track-statistics-for-streaming-spilling-20181219.patch.gz (application/gzip, 4.2K) download
0008-BUGFIX-set-final_lsn-for-subxacts-before-cl-20181219.patch.gz (application/gzip, 635B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2018-12-19 09:58 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Alexey Kondratov @ 2018-12-19 09:58 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Tomas,
> I'm a bit confused by the changes to TAP tests. Per the patch summary,
> some .pl files get renamed (nor sure why), a new one is added, etc.
I added new tap test case, streaming=true option inside old stream_*
ones and incremented streaming tests number (+2) because of the
collision between 009_matviews.pl / 009_stream_simple.pl and
010_truncate.pl / 010_stream_subxact.pl. At least in the previous
version of the patch they were under the same numbers. Nothing special,
but for simplicity, please, find attached my new tap test separately.
> So
> I've instead enabled streaming subscriptions in all tests, which with
> this patch produces two failures:
>
> Test Summary Report
> -------------------
> t/004_sync.pl (Wstat: 7424 Tests: 1 Failed: 0)
> Non-zero exit status: 29
> Parse errors: Bad plan. You planned 7 tests but ran 1.
> t/011_stream_ddl.pl (Wstat: 256 Tests: 2 Failed: 1)
> Failed test: 2
> Non-zero exit status: 1
>
> So yeah, there's more stuff to fix. But I can't directly apply your
> fixes because the updated patches are somewhat different.
Fixes should apply clearly to the previous version of your patch. Also,
I am not sure, that it is a good idea to simply enable streaming
subscriptions in all tests (e.g. pre streaming patch t/004_sync.pl),
since then they do not hit not streaming code.
>>> Interesting. Any idea where does the extra overhead in this particular
>>> case come from? It's hard to deduce that from the single flame graph,
>>> when I don't have anything to compare it with (i.e. the flame graph for
>>> the "normal" case).
>> I guess that bottleneck is in disk operations. You can check
>> logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
>> writes (~26%) take around 35% of CPU time in summary. To compare,
>> please, see attached flame graph for the following transaction:
>>
>> INSERT INTO large_text
>> SELECT (SELECT string_agg('x', ',')
>> FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>>
>> Execution Time: 44519.816 ms
>> Time: 98333,642 ms (01:38,334)
>>
>> where disk IO is only ~7-8% in total. So we get very roughly the same
>> ~x4-5 performance drop here. JFYI, I am using a machine with SSD for tests.
>>
>> Therefore, probably you may write changes on receiver in bigger chunks,
>> not each change separately.
>>
> Possibly, I/O is certainly a possible culprit, although we should be
> using buffered I/O and there certainly are not any fsyncs here. So I'm
> not sure why would it be cheaper to do the writes in batches.
>
> BTW does this mean you see the overhead on the apply side? Or are you
> running this on a single machine, and it's difficult to decide?
I run this on a single machine, but walsender and worker are utilizing
almost 100% of CPU per each process all the time, and at apply side I/O
syscalls take about 1/3 of CPU time. Though I am still not sure, but for
me this result somehow links performance drop with problems at receiver
side.
Writing in batches was just a hypothesis and to validate it I have
performed test with large txn, but consisting of a smaller number of
wide rows. This test does not exhibit any significant performance drop,
while it was streamed too. So it seems to be valid. Anyway, I do not
have other reasonable ideas beside that right now.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
0xx_stream_tough_ddl.pl (application/x-perl, 3.7K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-01-14 18:23 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
1 sibling, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-01-14 18:23 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi,
Attached is an updated patch series, merging fixes and changes to TAP
tests proposed by Alexey. I've merged the fixes into the appropriate
patches, and I've kept the TAP changes / new tests as separate patches
towards the end of the series.
I'm a bit unhappy with two aspects of the current patch series:
1) We now track schema changes in two ways - using the pre-existing
schema_sent flag in RelationSyncEntry, and the (newly added) flag in
ReorderBuffer. While those options are used for regular vs. streamed
transactions, fundamentally it's the same thing and so having two
competing ways seems like a bad idea. Not sure what's the best way to
resolve this, though.
2) We've removed quite a few asserts, particularly ensuring sanity of
cmin/cmax values. To some extent that's expected, because by allowing
decoding of in-progress transactions relaxes some of those rules. But
I'd be much happier if some of those asserts could be reinstated, even
if only in a weaker form.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_work_mem-to-limit-ReorderBuffer-20190114.patch.gz (application/gzip, 9.9K) download
0002-Immediately-WAL-log-assignments-20190114.patch.gz (application/gzip, 6.4K) download
0003-Issue-individual-invalidations-with-wal_lev-20190114.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-me-20190114.patch.gz (application/gzip, 6.1K) download
0005-Implement-streaming-mode-in-ReorderBuffer-20190114.patch.gz (application/gzip, 12.8K) download
0006-Add-support-for-streaming-to-built-in-repli-20190114.patch.gz (application/gzip, 19.8K) download
0007-Track-statistics-for-streaming-spilling-20190114.patch.gz (application/gzip, 4.2K) download
0008-Enable-streaming-for-all-subscription-TAP-t-20190114.patch.gz (application/gzip, 1.8K) download
0009-BUGFIX-set-final_lsn-for-subxacts-before-cl-20190114.patch.gz (application/gzip, 639B) download
0010-Add-TAP-test-for-streaming-vs.-DDL-20190114.patch.gz (application/gzip, 1.6K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-02-04 05:51 Michael Paquier <michael@paquier.xyz>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Michael Paquier @ 2019-02-04 05:51 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
On Mon, Jan 14, 2019 at 07:23:31PM +0100, Tomas Vondra wrote:
> Attached is an updated patch series, merging fixes and changes to TAP
> tests proposed by Alexey. I've merged the fixes into the appropriate
> patches, and I've kept the TAP changes / new tests as separate patches
> towards the end of the series.
Patch 4 of the latest set fails to apply, so I have moved the patch to
next CF, waiting on author.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-02-04 13:49 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Alexey Kondratov @ 2019-02-04 13:49 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Tomas,
On 14.01.2019 21:23, Tomas Vondra wrote:
> Attached is an updated patch series, merging fixes and changes to TAP
> tests proposed by Alexey. I've merged the fixes into the appropriate
> patches, and I've kept the TAP changes / new tests as separate patches
> towards the end of the series.
I had problems applying this patch along with 2pc streaming one to the
current master, but everything applied well on 97c39498e5. Regression
tests pass. What I personally do not like in the current TAP tests set
is that you have added "WITH (streaming=on)" to all tests including old
non-streaming ones. It seems unclear, which mechanism is tested there:
streaming, but those transactions probably do not hit memory limit, so
it depends on default server parameters; or non-streaming, but then what
is the need for (streaming=on)? I would prefer to add (streaming=on)
only to the new tests, where it is clearly necessary.
> I'm a bit unhappy with two aspects of the current patch series:
>
> 1) We now track schema changes in two ways - using the pre-existing
> schema_sent flag in RelationSyncEntry, and the (newly added) flag in
> ReorderBuffer. While those options are used for regular vs. streamed
> transactions, fundamentally it's the same thing and so having two
> competing ways seems like a bad idea. Not sure what's the best way to
> resolve this, though.
Yes, sure, when I have found problems with streaming of extensive DDL, I
added new flag in the simplest way, and it worked. Now, old schema_sent
flag is per relation based, while the new one - is_schema_sent - is per
top-level transaction based. If I get it correctly, the former seems to
be more thrifty, since new schema is sent only if we are streaming
change for relation, whose schema is outdated. In contrast, in the
latter case we will send new schema even if there will be no new changes
which belong to this relation.
I guess, it would be better to stick to the old behavior. I will try to
investigate how to better use it in the streaming mode as well.
> 2) We've removed quite a few asserts, particularly ensuring sanity of
> cmin/cmax values. To some extent that's expected, because by allowing
> decoding of in-progress transactions relaxes some of those rules. But
> I'd be much happier if some of those asserts could be reinstated, even
> if only in a weaker form.
Asserts have been removed from two places: (1)
HeapTupleSatisfiesHistoricMVCC, which seems inevitable, since we are
touching the essence of the MVCC visibility rules, when trying to decode
an in-progress transaction, and (2) ReorderBufferBuildTupleCidHash,
which is probably not related directly to the topic of the ongoing
patch, since Arseny Sher faced the same issue with simple repetitive DDL
decoding [1] recently.
Not many, but I agree, that replacing them with some softer asserts
would be better, than just removing, especially point 1).
[1] https://www.postgresql.org/message-id/flat/874l9p8hyw.fsf%40ars-thinkpad
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-08-28 17:17 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
1 sibling, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-08-28 17:17 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Hi Tomas,
>>>> Interesting. Any idea where does the extra overhead in this particular
>>>> case come from? It's hard to deduce that from the single flame graph,
>>>> when I don't have anything to compare it with (i.e. the flame graph
>>>> for
>>>> the "normal" case).
>>> I guess that bottleneck is in disk operations. You can check
>>> logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
>>> writes (~26%) take around 35% of CPU time in summary. To compare,
>>> please, see attached flame graph for the following transaction:
>>>
>>> INSERT INTO large_text
>>> SELECT (SELECT string_agg('x', ',')
>>> FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>>>
>>> Execution Time: 44519.816 ms
>>> Time: 98333,642 ms (01:38,334)
>>>
>>> where disk IO is only ~7-8% in total. So we get very roughly the same
>>> ~x4-5 performance drop here. JFYI, I am using a machine with SSD for
>>> tests.
>>>
>>> Therefore, probably you may write changes on receiver in bigger chunks,
>>> not each change separately.
>>>
>> Possibly, I/O is certainly a possible culprit, although we should be
>> using buffered I/O and there certainly are not any fsyncs here. So I'm
>> not sure why would it be cheaper to do the writes in batches.
>>
>> BTW does this mean you see the overhead on the apply side? Or are you
>> running this on a single machine, and it's difficult to decide?
>
> I run this on a single machine, but walsender and worker are utilizing
> almost 100% of CPU per each process all the time, and at apply side
> I/O syscalls take about 1/3 of CPU time. Though I am still not sure,
> but for me this result somehow links performance drop with problems at
> receiver side.
>
> Writing in batches was just a hypothesis and to validate it I have
> performed test with large txn, but consisting of a smaller number of
> wide rows. This test does not exhibit any significant performance
> drop, while it was streamed too. So it seems to be valid. Anyway, I do
> not have other reasonable ideas beside that right now.
I've checked recently this patch again and tried to elaborate it in
terms of performance. As a result I've implemented a new POC version of
the applier (attached). Almost everything in streaming logic stayed
intact, but apply worker is significantly different.
As I wrote earlier I still claim, that spilling changes on disk at the
applier side adds additional overhead, but it is possible to get rid of
it. In my additional patch I do the following:
1) Maintain a pool of additional background workers (bgworkers), that
are connected with main logical apply worker via shm_mq's. Each worker
is dedicated to the processing of specific streamed transaction.
2) When we receive a streamed change for some transaction, we check
whether there is an existing dedicated bgworker in HTAB (xid ->
bgworker), or there are some in the idle list, or spawn a new one.
3) We pass all changes (between STREAM START/STOP) to that bgworker via
shm_mq_send without intermediate waiting. However, we wait for bgworker
to apply the entire changes chunk at STREAM STOP, since we don't want
transactions reordering.
4) When transaction is commited/aborted worker is being added to the
idle list and is waiting for reassigning message.
5) I have used the same machinery with apply_dispatch in bgworkers,
since most of actions are practically very similar.
Thus, we do not spill anything at the applier side, so transaction
changes are processed by bgworkers as normal backends do. In the same
time, changes processing is strictly serial, which prevents transactions
reordering and possible conflicts/anomalies. Even though we trade off
performance in favor of stability the result is rather impressive. I
have used a similar query for testing as before:
EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1, num2, num3)
SELECT round(random()*10), random(), random()*142
FROM generate_series(1, 1000000) s(i);
with 1kk (1000000), 3kk and 5kk rows; logical_work_mem = 64MB and
synchronous_standby_names = 'FIRST 1 (large_sub)'. Table schema is
following:
CREATE TABLE large_test (
id serial primary key,
num1 bigint,
num2 double precision,
num3 double precision
);
Here are the results:
-------------------------------------------------------------------
| N | Time on master, sec | Total xact time, sec | Ratio |
-------------------------------------------------------------------
| On commit (master, v13) |
-------------------------------------------------------------------
| 1kk | 6.5 | 17.6 | x2.74 |
-------------------------------------------------------------------
| 3kk | 21 | 55.4 | x2.64 |
-------------------------------------------------------------------
| 5kk | 38.3 | 91.5 | x2.39 |
-------------------------------------------------------------------
| Stream + spill |
-------------------------------------------------------------------
| 1kk | 5.9 | 18 | x3 |
-------------------------------------------------------------------
| 3kk | 19.5 | 52.4 | x2.7 |
-------------------------------------------------------------------
| 5kk | 33.3 | 86.7 | x2.86 |
-------------------------------------------------------------------
| Stream + BGW pool |
-------------------------------------------------------------------
| 1kk | 6 | 12 | x2 |
-------------------------------------------------------------------
| 3kk | 18.5 | 30.5 | x1.65 |
-------------------------------------------------------------------
| 5kk | 35.6 | 53.9 | x1.51 |
-------------------------------------------------------------------
It seems that overhead added by synchronous replica is lower by 2-3
times compared with Postgres master and streaming with spilling.
Therefore, the original patch eliminated delay before large transaction
processing start by sender, while this additional patch speeds up the
applier side.
Although the overall speed up is surely measurable, there is a room for
improvements yet:
1) Currently bgworkers are only spawned on demand without some initial
pool and never stopped. Maybe we should create a small pool on
replication start and offload some of idle bgworkers if they exceed some
limit?
2) Probably we can track somehow that incoming change has conflicts with
some of being processed xacts, so we can wait for specific bgworkers
only in that case?
3) Since the communication between main logical apply worker and each
bgworker from the pool is a 'single producer --- single consumer'
problem, then probably it is possible to wait and set/check flags
without locks, but using just atomics.
What do you think about this concept in general? Any concerns and
criticism are welcome!
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
P.S. This patch shloud be applicable to your last patch set. I would rebase it against master, but it depends on 2pc patch, that I don't know well enough.
Attachments:
0011-BGWorkers-pool-for-streamed-transactions-apply-witho.patch (text/x-patch, 59.9K) download
From 11c7549d2732f2f983d4548a81cd509dd7e41ec4 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Wed, 28 Aug 2019 15:26:50 +0300
Subject: [PATCH 11/11] BGWorkers pool for streamed transactions apply without
spilling on disk
---
src/backend/postmaster/bgworker.c | 3 +
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/logical/proto.c | 17 +-
src/backend/replication/logical/worker.c | 1780 +++++++++++-----------
src/include/pgstat.h | 1 +
src/include/replication/logicalproto.h | 4 +-
src/include/replication/logicalworker.h | 1 +
7 files changed, 933 insertions(+), 876 deletions(-)
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index f5db5a8c4a..6860df07ca 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "LogicalApplyBgwMain", LogicalApplyBgwMain
}
};
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index e5a4d147a7..b32994784f 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3637,6 +3637,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING:
event_name = "Hash/GrowBuckets/Reinserting";
break;
+ case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY:
+ event_name = "LogicalApplyWorkerReady";
+ break;
case WAIT_EVENT_LOGICAL_SYNC_DATA:
event_name = "LogicalSyncData";
break;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 4bec9fe8b5..954ce7343a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -789,14 +789,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
pq_sendint64(out, txn->commit_time);
}
-TransactionId
+void
logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
{
- TransactionId xid;
uint8 flags;
- xid = pq_getmsgint(in, 4);
-
/* read flags (unused for now) */
flags = pq_getmsgbyte(in);
@@ -807,8 +804,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
commit_data->commit_lsn = pq_getmsgint64(in);
commit_data->end_lsn = pq_getmsgint64(in);
commit_data->committime = pq_getmsgint64(in);
-
- return xid;
}
void
@@ -823,13 +818,3 @@ logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
pq_sendint32(out, xid);
pq_sendint32(out, subxid);
}
-
-void
-logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid)
-{
- Assert(xid && subxid);
-
- *xid = pq_getmsgint(in, 4);
- *subxid = pq_getmsgint(in, 4);
-}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ca632b7dc4..dc6c895fca 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -92,11 +92,16 @@
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+// #include "storage/condition_variable.h"
+#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/shm_mq.h"
+#include "storage/shm_toc.h"
+#include "storage/spin.h"
#include "tcop/tcopprot.h"
@@ -115,6 +120,54 @@
#include "utils/syscache.h"
#define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */
+#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change
+
+typedef struct ParallelState
+{
+ slock_t mutex;
+ // ConditionVariable cv;
+ bool attached;
+ bool ready;
+ bool finished;
+ Oid database_id;
+ Oid authenticated_user_id;
+ Oid subid;
+ Oid stream_xid;
+ uint32 n;
+} ParallelState;
+
+typedef struct WorkerState
+{
+ TransactionId xid;
+ BackgroundWorkerHandle *handle;
+ shm_mq_handle *mq_handle;
+ dsm_segment *dsm_seg;
+ ParallelState volatile *pstate;
+} WorkerState;
+
+/* Apply workers hash table (initialized on first use) */
+static HTAB *ApplyWorkersHash = NULL;
+static WorkerState **ApplyWorkersIdleList = NULL;
+static uint32 pool_size = 10; /* MaxConnections default? */
+static uint32 nworkers = 0;
+static uint32 nfreeworkers = 0;
+
+/* Fields valid only for apply background workers */
+bool isLogicalApplyWorker = false;
+volatile ParallelState *MyParallelState = NULL;
+
+/* Worker setup and interactions */
+static void setup_dsm(WorkerState *wstate);
+static void setup_background_worker(WorkerState *wstate);
+static void cleanup_background_worker(dsm_segment *seg, Datum arg);
+static void handle_sigterm(SIGNAL_ARGS);
+
+static bool check_worker_status(WorkerState *wstate);
+static void wait_for_worker(WorkerState *wstate);
+static void wait_for_worker_to_finish(WorkerState *wstate);
+
+static WorkerState * find_or_start_worker(TransactionId xid, bool start);
+static void stop_worker(WorkerState *wstate);
typedef struct FlushPosition
{
@@ -143,47 +196,13 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
-/* fields valid only when processing streamed transaction */
+/* Fields valid only when processing streamed transaction */
bool in_streamed_transaction = false;
static TransactionId stream_xid = InvalidTransactionId;
-static int stream_fd = -1;
-
-typedef struct SubXactInfo
-{
- TransactionId xid; /* XID of the subxact */
- off_t offset; /* offset in the file */
-} SubXactInfo;
-
-static uint32 nsubxacts = 0;
-static uint32 nsubxacts_max = 0;
-static SubXactInfo * subxacts = NULL;
-static TransactionId subxact_last = InvalidTransactionId;
-
-static void subxact_filename(char *path, Oid subid, TransactionId xid);
-static void changes_filename(char *path, Oid subid, TransactionId xid);
-
-/*
- * Information about subtransactions of a given toplevel transaction.
- */
-static void subxact_info_write(Oid subid, TransactionId xid);
-static void subxact_info_read(Oid subid, TransactionId xid);
-static void subxact_info_add(TransactionId xid);
-
-/*
- * Serialize and deserialize changes for a toplevel transaction.
- */
-static void stream_cleanup_files(Oid subid, TransactionId xid);
-static void stream_open_file(Oid subid, TransactionId xid, bool first);
-static void stream_write_change(char action, StringInfo s);
-static void stream_close_file(void);
-
-/*
- * Array of serialized XIDs.
- */
-static int nxids = 0;
-static int maxnxids = 0;
-static TransactionId *xids = NULL;
+static TransactionId current_xid = InvalidTransactionId;
+static TransactionId prev_xid = InvalidTransactionId;
+static uint32 nchanges = 0;
static bool handle_streamed_transaction(const char action, StringInfo s);
@@ -199,6 +218,16 @@ static volatile sig_atomic_t got_SIGHUP = false;
/* prototype needed because of stream_commit */
static void apply_dispatch(StringInfo s);
+// /* Debug only */
+// static void
+// iter_sleep(int seconds)
+// {
+// for (int i = 0; i < seconds; i++)
+// {
+// pg_usleep(1 * 1000L * 1000L);
+// }
+// }
+
/*
* Should this worker apply changes for given relation.
*
@@ -250,6 +279,107 @@ ensure_transaction(void)
return true;
}
+/*
+ * Look up worker inside ApplyWorkersHash for requested xid.
+ * Throw error if not found or start a new one if start=true is passed.
+ */
+static WorkerState *
+find_or_start_worker(TransactionId xid, bool start)
+{
+ bool found;
+ WorkerState *entry = NULL;
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* First time through, initialize apply workers hashtable */
+ if (ApplyWorkersHash == NULL)
+ {
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(WorkerState);
+ ctl.hcxt = ApplyContext; /* Allocate ApplyWorkersHash in the ApplyContext */
+ ApplyWorkersHash = hash_create("logical apply workers hash", 8,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ }
+
+ Assert(ApplyWorkersHash != NULL);
+
+ /*
+ * Find entry for requested transaction.
+ */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found);
+
+ if (!found && start)
+ {
+ /* If there is at least one worker in the idle list, then take one. */
+ if (nfreeworkers > 0)
+ {
+ char action = 'R';
+
+ Assert(ApplyWorkersIdleList != NULL);
+
+ entry = ApplyWorkersIdleList[nfreeworkers - 1];
+ if (!hash_update_hash_key(ApplyWorkersHash,
+ (void *) entry,
+ (void *) &xid))
+ elog(ERROR, "could not reassign apply worker #%u entry from xid %u to xid %u",
+ entry->pstate->n, entry->xid, xid);
+
+ entry->xid = xid;
+ entry->pstate->finished = false;
+ entry->pstate->stream_xid = xid;
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+
+ ApplyWorkersIdleList[--nfreeworkers] = NULL;
+ }
+ else
+ {
+ /* No entry in hash and no idle workers. Create a new one. */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found);
+ entry->xid = xid;
+ setup_background_worker(entry);
+
+ if (nworkers == pool_size)
+ {
+ ApplyWorkersIdleList = repalloc(ApplyWorkersIdleList, pool_size + 10);
+ pool_size += 10;
+ }
+ }
+ }
+ else if (!found && !start)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not find logical apply worker for xid %u", xid)));
+ else
+ elog(DEBUG5, "there is an existing logical apply worker for xid %u", xid);
+
+ Assert(entry != NULL);
+
+ return entry;
+}
+
+/*
+ * Gracefully teardown apply worker.
+ */
+static void
+stop_worker(WorkerState *wstate)
+{
+ /*
+ * Sending zero-length data to worker in order to stop it.
+ */
+ shm_mq_send(wstate->mq_handle, 0, NULL, false);
+
+ elog(LOG, "detaching DSM of apply worker #%u for xid %u",
+ wstate->pstate->n, wstate->xid);
+ dsm_detach(wstate->dsm_seg);
+
+ /* Delete worker entry */
+ (void) hash_search(ApplyWorkersHash, &wstate->xid, HASH_REMOVE, NULL);
+}
+
/*
* Handle streamed transactions.
*
@@ -262,12 +392,12 @@ static bool
handle_streamed_transaction(const char action, StringInfo s)
{
TransactionId xid;
+ WorkerState *entry;
/* not in streaming mode */
- if (!in_streamed_transaction)
+ if (!in_streamed_transaction || isLogicalApplyWorker)
return false;
- Assert(stream_fd != -1);
Assert(TransactionIdIsValid(stream_xid));
/*
@@ -278,11 +408,16 @@ handle_streamed_transaction(const char action, StringInfo s)
Assert(TransactionIdIsValid(xid));
- /* Add the new subxact to the array (unless already there). */
- subxact_info_add(xid);
+ /*
+ * Find worker for requested xid.
+ */
+ entry = find_or_start_worker(stream_xid, false);
- /* write the change to the current file */
- stream_write_change(action, s);
+ // elog(LOG, "sending message of length=%d and raw=%s, action=%s", s->len, s->data, (char *) &action);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
+ nchanges += 1;
+
+ // iter_sleep(3600);
return true;
}
@@ -643,7 +778,8 @@ apply_handle_origin(StringInfo s)
static void
apply_handle_stream_start(StringInfo s)
{
- bool first_segment;
+ bool first_segment;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
@@ -652,17 +788,16 @@ apply_handle_stream_start(StringInfo s)
/* extract XID of the top-level transaction */
stream_xid = logicalrep_read_stream_start(s, &first_segment);
+ nchanges = 0;
- /* open the spool file for this transaction */
- stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, true);
- /*
- * if this is not the first segment, open existing file
- *
- * XXX Note that the cleanup is performed by stream_open_file.
- */
- if (!first_segment)
- subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+ SpinLockAcquire(&entry->pstate->mutex);
+ entry->pstate->ready = false;
+ SpinLockRelease(&entry->pstate->mutex);
+
+ elog(LOG, "starting streaming of xid %u", stream_xid);
pgstat_report_activity(STATE_RUNNING, NULL);
}
@@ -673,16 +808,19 @@ apply_handle_stream_start(StringInfo s)
static void
apply_handle_stream_stop(StringInfo s)
{
+ WorkerState *entry;
+ char action = 'E';
+
Assert(in_streamed_transaction);
- /*
- * Close the file with serialized changes, and serialize information about
- * subxacts for the toplevel transaction.
- */
- subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
- stream_close_file();
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
+
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ wait_for_worker(entry);
in_streamed_transaction = false;
+ elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges);
pgstat_report_activity(STATE_IDLE, NULL);
}
@@ -695,96 +833,67 @@ apply_handle_stream_abort(StringInfo s)
{
TransactionId xid;
TransactionId subxid;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
- logicalrep_read_stream_abort(s, &xid, &subxid);
-
- /*
- * If the two XIDs are the same, it's in fact abort of toplevel xact, so
- * just delete the files with serialized info.
- */
- if (xid == subxid)
+ if(isLogicalApplyWorker)
{
- char path[MAXPGPATH];
+ subxid = pq_getmsgint(s, 4);
- /*
- * XXX Maybe this should be an error instead? Can we receive abort for
- * a toplevel transaction we haven't received?
- */
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u",
+ MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId())));
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ if (subxid == stream_xid)
+ AbortCurrentTransaction();
+ else
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", subxid);
- if (unlink(path) < 0)
- ereport(ERROR,
+ ereport(LOG,
(errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname)));
- subxact_filename(path, MyLogicalRepWorker->subid, xid);
-
- if (unlink(path) < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ RollbackToSavepoint(spname);
+ CommitTransactionCommand();
+ // RollbackAndReleaseCurrentSubTransaction();
- return;
+ pfree(spname);
+ }
}
else
{
- /*
- * OK, so it's a subxact. We need to read the subxact file for the
- * toplevel transaction, determine the offset tracked for the subxact,
- * and truncate the file with changes. We also remove the subxacts
- * with higher offsets (or rather higher XIDs).
- *
- * We intentionally scan the array from the tail, because we're likely
- * aborting a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- *
- * XXX Or perhaps we can rely on the aborts to arrive in the reverse
- * order, i.e. from the inner-most subxact (when nested)? In which
- * case we could simply check the last element.
- */
+ xid = pq_getmsgint(s, 4);
+ subxid = pq_getmsgint(s, 4);
- int64 i;
- int64 subidx;
- bool found = false;
- char path[MAXPGPATH];
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
- subidx = -1;
- subxact_info_read(MyLogicalRepWorker->subid, xid);
+ elog(LOG, "processing abort request of streamed transaction xid %u, subxid %u",
+ xid, subxid);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- /* FIXME optimize the search by bsearch on sorted data */
- for (i = nsubxacts; i > 0; i--)
+ if (subxid == stream_xid)
{
- if (subxacts[i - 1].xid == subxid)
- {
- subidx = (i - 1);
- found = true;
- break;
- }
- }
-
- /* We should not receive aborts for unknown subtransactions. */
- Assert(found);
+ char action = 'F';
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ // shm_mq_send(entry->mq_handle, 0, NULL, false);
- /* OK, truncate the file at the right offset. */
- Assert((subidx >= 0) && (subidx < nsubxacts));
+ wait_for_worker_to_finish(entry);
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
- if (truncate(path, subxacts[subidx].offset))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not truncate file \"%s\": %m", path)));
+ // elog(LOG, "detaching DSM of apply worker for xid=%u\n", entry->xid);
+ // dsm_detach(entry->dsm_seg);
- /* discard the subxacts added later */
- nsubxacts = subidx;
-
- /* write the updated subxact list */
- subxact_info_write(MyLogicalRepWorker->subid, xid);
+ // /* Delete worker entry */
+ // (void) hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL);
+ }
}
}
@@ -794,159 +903,56 @@ apply_handle_stream_abort(StringInfo s)
static void
apply_handle_stream_commit(StringInfo s)
{
- int fd;
TransactionId xid;
- StringInfoData s2;
- int nchanges;
-
- char path[MAXPGPATH];
- char *buffer = NULL;
+ WorkerState *entry;
LogicalRepCommitData commit_data;
- MemoryContext oldcxt;
-
- Assert(!in_streamed_transaction);
-
- xid = logicalrep_read_stream_commit(s, &commit_data);
-
- elog(DEBUG1, "received commit for streamed transaction %u", xid);
-
- /* open the spool file for the committed transaction */
- changes_filename(path, MyLogicalRepWorker->subid, xid);
-
- elog(DEBUG1, "replaying changes from file '%s'", path);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
+ if (isLogicalApplyWorker)
{
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- }
-
- /* XXX Should this be allocated in another memory context? */
+ // logicalrep_read_stream_commit(s, &commit_data);
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- buffer = palloc(8192);
- initStringInfo(&s2);
-
- MemoryContextSwitchTo(oldcxt);
-
- ensure_transaction();
-
- /*
- * Make sure the handle apply_dispatch methods are aware we're in a remote
- * transaction.
- */
- in_remote_transaction = true;
- pgstat_report_activity(STATE_RUNNING, NULL);
-
- /*
- * Read the entries one by one and pass them through the same logic as in
- * apply_dispatch.
- */
- nchanges = 0;
- while (true)
+ CommitTransactionCommand();
+ }
+ else
{
- int nbytes;
- int len;
-
- /* read length of the on-disk record */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- nbytes = read(fd, &len, sizeof(len));
- pgstat_report_wait_end();
-
- /* have we reached end of the file? */
- if (nbytes == 0)
- break;
-
- /* do we have a correct length? */
- if (nbytes != sizeof(len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
-
- Assert(len > 0);
+ char action = 'F';
- /* make sure we have sufficiently large buffer */
- buffer = repalloc(buffer, len);
-
- /* and finally read the data into the buffer */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- if (read(fd, buffer, len) != len)
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
- pgstat_report_wait_end();
+ Assert(!in_streamed_transaction);
- /* copy the buffer to the stringinfo and call apply_dispatch */
- resetStringInfo(&s2);
- appendBinaryStringInfo(&s2, buffer, len);
+ xid = pq_getmsgint(s, 4);
+ logicalrep_read_stream_commit(s, &commit_data);
- /* Ensure we are reading the data into our memory context. */
- oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
- apply_dispatch(&s2);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(xid, false);
- MemoryContextReset(ApplyMessageContext);
+ /* Send commit message */
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- MemoryContextSwitchTo(oldcxt);
+ /* Notify worker, that we are done with this xact */
+ shm_mq_send(entry->mq_handle, 1, &action, false);
- nchanges++;
+ wait_for_worker_to_finish(entry);
- if (nchanges % 1000 == 0)
- elog(DEBUG1, "replayed %d changes from file '%s'",
- nchanges, path);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
/*
- * send feedback to upstream
- *
- * XXX Probably should send a valid LSN. But which one?
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
*/
- send_feedback(InvalidXLogRecPtr, false, false);
- }
-
- CloseTransientFile(fd);
-
- /*
- * Update origin state so we can restart streaming from correct
- * position in case of crash.
- */
- replorigin_session_origin_lsn = commit_data.end_lsn;
- replorigin_session_origin_timestamp = commit_data.committime;
-
- CommitTransactionCommand();
- pgstat_report_stat(false);
-
- store_flush_position(commit_data.end_lsn);
-
- elog(DEBUG1, "replayed %d (all) changes from file '%s'",
- nchanges, path);
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
- in_remote_transaction = false;
- pgstat_report_activity(STATE_IDLE, NULL);
+ pgstat_report_stat(false);
- /* unlink the files with serialized changes and subxact info */
- stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ store_flush_position(commit_data.end_lsn);
- pfree(buffer);
- pfree(s2.data);
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
}
/*
@@ -965,6 +971,8 @@ apply_handle_relation(StringInfo s)
if (handle_streamed_transaction('R', s))
return;
+ // iter_sleep(3600);
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -1407,6 +1415,35 @@ apply_dispatch(StringInfo s)
{
char action = pq_getmsgbyte(s);
+ if (isLogicalApplyWorker)
+ {
+ /*
+ * Inside logical apply worker we can figure out that new subtransaction
+ * was started if new change arrived with different xid. In that case we
+ * can define named savepoint, so that we were able to commit/rollback it
+ * separately later.
+ */
+ current_xid = pq_getmsgint(s, 4);
+
+ if (prev_xid == InvalidTransactionId)
+ prev_xid = current_xid;
+ else if (current_xid != prev_xid && current_xid != stream_xid)
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", current_xid);
+
+ elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname);
+
+ DefineSavepoint(spname);
+ CommitTransactionCommand();
+ // BeginInternalSubTransaction(NULL);
+ }
+
+ prev_xid = current_xid;
+ }
+ // else
+ // elog(LOG, "Logical worker: applying dispatch for action=%s", (char *) &action);
+
switch (action)
{
/* BEGIN */
@@ -1435,6 +1472,7 @@ apply_dispatch(StringInfo s)
break;
/* RELATION */
case 'R':
+ // elog(LOG, "%s worker: applying dispatch for action=R", isLogicalApplyWorker ? "Apply" : "Logical");
apply_handle_relation(s);
break;
/* TYPE */
@@ -1565,12 +1603,18 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
static void
worker_onexit(int code, Datum arg)
{
- int i;
+ HASH_SEQ_STATUS status;
+ WorkerState *entry;
- elog(LOG, "cleanup files for %d transactions", nxids);
-
- for (i = nxids-1; i >= 0; i--)
- stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+ if (ApplyWorkersHash != NULL)
+ {
+ hash_seq_init(&status, ApplyWorkersHash);
+ while ((entry = (WorkerState *) hash_seq_search(&status)) != NULL)
+ {
+ stop_worker(entry);
+ }
+ hash_seq_term(&status);
+ }
}
/*
@@ -1593,6 +1637,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
+ ApplyWorkersIdleList = palloc(sizeof(WorkerState *) * pool_size);
+
for (;;)
{
pgsocket fd = PGINVALID_SOCKET;
@@ -1904,8 +1950,9 @@ maybe_reread_subscription(void)
Subscription *newsub;
bool started_tx = false;
+ // TODO Probably we have to handle subscription reread in apply workers too.
/* When cache state is valid there is nothing to do here. */
- if (MySubscriptionValid)
+ if (MySubscriptionValid || isLogicalApplyWorker)
return;
/* This function might be called inside or outside of transaction. */
@@ -2039,608 +2086,50 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
-/*
- * subxact_info_write
- * Store information about subxacts for a toplevel transaction.
- *
- * For each subxact we store offset of it's first change in the main file.
- * The file is always over-written as a whole, and we also include CRC32C
- * checksum of the information.
- *
- * XXX We should only store subxacts that were not aborted yet.
- *
- * XXX Maybe we should only include the checksum when the cluster is
- * initialized with checksums?
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- */
+/* SIGHUP: set flag to reload configuration at next convenient time */
static void
-subxact_info_write(Oid subid, TransactionId xid)
+logicalrep_worker_sighup(SIGNAL_ARGS)
{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- Size len;
-
- Assert(TransactionIdIsValid(xid));
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create file \"%s\": %m",
- path)));
- return;
- }
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* compute the checksum */
- INIT_CRC32C(checksum);
- COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum, (char *) subxacts, len);
- FIN_CRC32C(checksum);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
-
- if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
+ int save_errno = errno;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ got_SIGHUP = true;
- if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ errno = save_errno;
+}
- if ((len > 0) && (write(fd, subxacts, len) != len))
- {
- int save_errno = errno;
+/* Logical Replication Apply worker entry point */
+void
+ApplyWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ MemoryContext oldctx;
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos;
+ char *myslotname;
+ WalRcvStreamOptions options;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
- pgstat_report_wait_end();
+ /* Setup signal handling */
+ pqsignal(SIGHUP, logicalrep_worker_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
/*
- * We don't need to fsync or anything, as we'll recreate the files after a
- * crash from scratch. So just close the file.
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
*/
- CloseTransientFile(fd);
- /*
- * But we free the memory allocated for subxact info. There might be one
- * exceptional transaction with many subxacts, and we don't want to keep
- * the memory allocated forewer.
- *
- */
- if (subxacts)
- pfree(subxacts);
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
- subxacts = NULL;
- subxact_last = InvalidTransactionId;
- nsubxacts = 0;
- nsubxacts_max = 0;
-}
-
-/*
- * subxact_info_read
- * Restore information about subxacts of a streamed transaction.
- *
- * Read information about subxacts into the global variables, and while
- * reading the information verify the checksum.
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_read(Oid subid, TransactionId xid)
-{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- uint32 checksum_new;
- Size len;
- MemoryContext oldctx;
-
- Assert(TransactionIdIsValid(xid));
- Assert(!subxacts);
- Assert(nsubxacts == 0);
- Assert(nsubxacts_max == 0);
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- /* read the checksum */
- if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- /* read number of subxact items */
- if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* we keep the maximum as a power of 2 */
- nsubxacts_max = 1 << my_log2(nsubxacts);
-
- /* subxacts are long-lived */
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- if ((len > 0) && ((read(fd, subxacts, len)) != len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- /* recompute the checksum */
- INIT_CRC32C(checksum_new);
- COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum_new, (char *) subxacts, len);
- FIN_CRC32C(checksum_new);
-
- if (checksum_new != checksum)
- ereport(ERROR,
- (errmsg("checksum failure when reading subxacts")));
-
- CloseTransientFile(fd);
-}
-
-/*
- * subxact_info_add
- * Add information about a subxact (offset in the main file).
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_add(TransactionId xid)
-{
- int64 i;
-
- /*
- * If the XID matches the toplevel transaction, we don't want to add it.
- */
- if (stream_xid == xid)
- return;
-
- /*
- * In most cases we're checking the same subxact as we've already seen in
- * the last call, so make ure just ignore it (this change comes later).
- */
- if (subxact_last == xid)
- return;
-
- /* OK, remember we're processing this XID. */
- subxact_last = xid;
-
- /*
- * Check if the transaction is already present in the array of subxact. We
- * intentionally scan the array from the tail, because we're likely adding
- * a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- */
- for (i = nsubxacts; i > 0; i--)
- {
- /* found, so we're done */
- if (subxacts[i - 1].xid == xid)
- return;
- }
-
- /* This is a new subxact, so we need to add it to the array. */
-
- if (nsubxacts == 0)
- {
- MemoryContext oldctx;
-
- nsubxacts_max = 128;
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
- }
- else if (nsubxacts == nsubxacts_max)
- {
- nsubxacts_max *= 2;
- subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
- }
-
- subxacts[nsubxacts].xid = xid;
- subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
-
- nsubxacts++;
-}
-
-/* format filename for file containing the info about subxacts */
-static void
-subxact_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
- tempdirpath, subid, xid);
-}
-
-/* format filename for file containing serialized changes */
-static void
-changes_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
- tempdirpath, subid, xid);
-}
-
-/*
- * stream_cleanup_files
- * Cleanup files for a subscription / toplevel transaction.
- *
- * Remove files with serialized changes and subxact info for a particular
- * toplevel transaction. Each subscription has a separate set of files.
- *
- * Note: The files may not exists, so handle ENOENT as non-error.
- *
- * TODO: Add missing_ok flag to specify in which cases it's OK not to
- * find the files, and when it's an error.
- */
-static void
-stream_cleanup_files(Oid subid, TransactionId xid)
-{
- int i;
- char path[MAXPGPATH];
- bool found = false;
-
- subxact_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- changes_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- /*
- * Cleanup the XID from the array - find the XID in the array and
- * remove it by shifting all the remaining elements. The array is
- * bound to be fairly small (maximum number of in-progress xacts,
- * so max_connections + max_prepared_transactions) so simply loop
- * through the array and find index of the XID. Then move the rest
- * of the array by one element to the left.
- *
- * Notice we also call this from stream_open_file for first segment
- * of each transaction, to deal with possible left-overs after a
- * crash, so it's entirely possible not to find the XID in the
- * array here. In that case we don't remove anything.
- *
- * XXX Perhaps it'd be better to handle this automatically after a
- * restart, instead of doing it over and over for each transaction.
- */
- for (i = 0; i < nxids; i++)
- {
- if (xids[i] == xid)
- {
- found = true;
- break;
- }
- }
-
- if (!found)
- return;
-
- /*
- * Move the last entry from the array to the place. We don't keep
- * the streamed transactions sorted or anything - we only expect
- * a few of them in progress (max_connections + max_prepared_xacts)
- * so linear search is just fine.
- */
- xids[i] = xids[nxids-1];
- nxids--;
-}
-
-/*
- * stream_open_file
- * Open file we'll use to serialize changes for a toplevel transaction.
- *
- * Open a file for streamed changes from a toplevel transaction identified
- * by stream_xid (global variable). If it's the first chunk of streamed
- * changes for this transaction, perform cleanup by removing existing
- * files after a possible previous crash.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_open_file(Oid subid, TransactionId xid, bool first_segment)
-{
- char path[MAXPGPATH];
- int flags;
-
- Assert(in_streamed_transaction);
- Assert(OidIsValid(subid));
- Assert(TransactionIdIsValid(xid));
- Assert(stream_fd == -1);
-
- /*
- * If this is the first segment for this transaction, try removing
- * existing files (if there are any, possibly after a crash).
- */
- if (first_segment)
- {
- MemoryContext oldcxt;
-
- /* XXX make sure there are no previous files for this transaction */
- stream_cleanup_files(subid, xid);
-
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- /*
- * We need to remember the XIDs we spilled to files, so that we can
- * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
- *
- * The number of XIDs we may need to track is fairly small, because
- * we can only stream toplevel xacts (so limited by max_connections
- * and max_prepared_transactions), and we only stream the large ones.
- * So we simply keep the XIDs in an unsorted array. If the number of
- * xacts gets large for some reason (e.g. very high max_connections),
- * a more elaborate approach might be better - e.g. sorted array, to
- * speed-up the lookups.
- */
- if (nxids == maxnxids) /* array of XIDs is full */
- {
- if (!xids)
- {
- maxnxids = 64;
- xids = palloc(maxnxids * sizeof(TransactionId));
- }
- else
- {
- maxnxids = 2 * maxnxids;
- xids = repalloc(xids, maxnxids * sizeof(TransactionId));
- }
- }
-
- xids[nxids++] = xid;
-
- MemoryContextSwitchTo(oldcxt);
- }
-
- changes_filename(path, subid, xid);
-
- elog(DEBUG1, "opening file '%s' for streamed changes", path);
-
- /*
- * If this is the first streamed segment, the file must not exist, so
- * make sure we're the ones creating it. Otherwise just open the file
- * for writing, in append mode.
- */
- if (first_segment)
- flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
- else
- flags = (O_WRONLY | O_APPEND | PG_BINARY);
-
- stream_fd = OpenTransientFile(path, flags);
-
- if (stream_fd < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
-}
-
-/*
- * stream_close_file
- * Close the currently open file with streamed changes.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_close_file(void)
-{
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- CloseTransientFile(stream_fd);
-
- stream_xid = InvalidTransactionId;
- stream_fd = -1;
-}
-
-/*
- * stream_write_change
- * Serialize a change to a file for the current toplevel transaction.
- *
- * The change is serialied in a simple format, with length (not including
- * the length), action code (identifying the message type) and message
- * contents (without the subxact TransactionId value).
- *
- * XXX The subxact file includes CRC32C of the contents. Maybe we should
- * include something like that here too, but doing so will not be as
- * straighforward, because we write the file in chunks.
- */
-static void
-stream_write_change(char action, StringInfo s)
-{
- int len;
-
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- /* total on-disk size, including the action type character */
- len = (s->len - s->cursor) + sizeof(char);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
-
- /* first write the size */
- if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* then the action */
- if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* and finally the remaining part of the buffer (after the XID) */
- len = (s->len - s->cursor);
-
- if (write(stream_fd, &s->data[s->cursor], len) != len)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- pgstat_report_wait_end();
-}
-
-/* SIGHUP: set flag to reload configuration at next convenient time */
-static void
-logicalrep_worker_sighup(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- got_SIGHUP = true;
-
- /* Waken anything waiting on the process latch */
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
-/* Logical Replication Apply worker entry point */
-void
-ApplyWorkerMain(Datum main_arg)
-{
- int worker_slot = DatumGetInt32(main_arg);
- MemoryContext oldctx;
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos;
- char *myslotname;
- WalRcvStreamOptions options;
-
- /* Attach to slot */
- logicalrep_worker_attach(worker_slot);
-
- /* Setup signal handling */
- pqsignal(SIGHUP, logicalrep_worker_sighup);
- pqsignal(SIGTERM, die);
- BackgroundWorkerUnblockSignals();
-
- /*
- * We don't currently need any ResourceOwner in a walreceiver process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
-
- /* Initialise stats to a sanish value */
- MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
- MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
-
- /* Load the libpq-specific functions */
- load_file("libpqwalreceiver", false);
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
/* Run as replica session replication role. */
SetConfigOption("session_replication_role", "replica",
@@ -2798,3 +2287,580 @@ IsLogicalWorker(void)
{
return MyLogicalRepWorker != NULL;
}
+
+/*
+ * Apply Background Worker main loop.
+ */
+void
+LogicalApplyBgwMain(Datum main_arg)
+{
+ volatile ParallelState *pst;
+
+ dsm_segment *seg;
+ shm_toc *toc;
+ PGPROC *registrant;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ shm_mq_result shmq_res;
+ // ConditionVariable cv;
+ LogicalRepWorker lrw;
+ MemoryContext oldcontext;
+
+ MemoryContextSwitchTo(TopMemoryContext);
+
+ /* Load the subscription into persistent memory context. */
+ ApplyContext = AllocSetContextCreate(TopMemoryContext,
+ "ApplyContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldcontext = MemoryContextSwitchTo(ApplyContext);
+
+ /*
+ * Init the ApplyMessageContext which we clean up after each replication
+ * protocol message.
+ */
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ isLogicalApplyWorker = true;
+
+ /*
+ * Establish signal handlers.
+ *
+ * We want CHECK_FOR_INTERRUPTS() to kill off this worker process just as
+ * it would a normal user backend. To make that happen, we establish a
+ * signal handler that is a stripped-down version of die().
+ */
+ pqsignal(SIGTERM, handle_sigterm);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Connect to the dynamic shared memory segment.
+ *
+ * The backend that registered this worker passed us the ID of a shared
+ * memory segment to which we must attach for further instructions. In
+ * order to attach to dynamic shared memory, we need a resource owner.
+ * Once we've mapped the segment in our address space, attach to the table
+ * of contents so we can locate the various data structures we'll need to
+ * find within the segment.
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker");
+ seg = dsm_attach(DatumGetInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("unable to map dynamic shared memory segment")));
+ toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg));
+ if (toc == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("bad magic number in dynamic shared memory segment")));
+
+ /*
+ * Acquire a worker number.
+ *
+ * By convention, the process registering this background worker should
+ * have stored the control structure at key 0. We look up that key to
+ * find it. Our worker number gives our identity: there may be just one
+ * worker involved in this parallel operation, or there may be many.
+ */
+ pst = shm_toc_lookup(toc, 0, false);
+ MyParallelState = pst;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->attached = true;
+ SpinLockRelease(&pst->mutex);
+
+ /*
+ * Attach to the message queue.
+ */
+ mq = shm_toc_lookup(toc, 1, false);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ /* Restore database connection. */
+ BackgroundWorkerInitializeConnectionByOid(pst->database_id,
+ pst->authenticated_user_id, 0);
+
+ /*
+ * Set the client encoding to the database encoding, since that is what
+ * the leader will expect.
+ */
+ SetClientEncoding(GetDatabaseEncoding());
+
+ lrw.subid = pst->subid;
+ MyLogicalRepWorker = &lrw;
+
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+ // PushActiveSnapshot(GetTransactionSnapshot());
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ /*
+ * Indicate that we're fully initialized and ready to begin the main part
+ * of the parallel operation.
+ *
+ * Once we signal that we're ready, the user backend is entitled to assume
+ * that our on_dsm_detach callbacks will fire before we disconnect from
+ * the shared memory segment and exit. Generally, that means we must have
+ * attached to all relevant dynamic shared memory data structures by now.
+ */
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ // cv = pst->cv;
+ // if (pst->workers_ready == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+ elog(LOG, "[Apply BGW #%u] started", pst->n);
+
+ registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ SetLatch(®istrant->procLatch);
+
+ for (;;)
+ {
+ void *data;
+ Size len;
+ StringInfoData s;
+ MemoryContext oldctx;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Ensure we are reading the data into our memory context. */
+ oldctx = MemoryContextSwitchTo(ApplyMessageContext);
+
+ shmq_res = shm_mq_receive(mqh, &len, &data, false);
+
+ if (shmq_res != SHM_MQ_SUCCESS)
+ break;
+
+ if (len == 0)
+ {
+ elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n);
+ break;
+ }
+ else
+ {
+ s.cursor = 0;
+ s.maxlen = -1;
+ s.data = (char *) data;
+ s.len = len;
+
+ /*
+ * We use first byte of message for additional communication between
+ * main Logical replication worker and Apply BGWorkers, so if it
+ * differs from 'w', then process it first.
+ */
+ switch (pq_getmsgbyte(&s))
+ {
+ /* Stream stop */
+ case 'E':
+ {
+ in_remote_transaction = false;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ SpinLockRelease(&pst->mutex);
+ SetLatch(®istrant->procLatch);
+
+ elog(LOG, "[Apply BGW #%u] ended processing streaming chunk, waiting on shm_mq_receive", pst->n);
+
+ continue;
+ }
+ /* Reassign to the new transaction */
+ case 'R':
+ {
+ elog(LOG, "[Apply BGW #%u] switching from processing xid %u to xid %u",
+ pst->n, stream_xid, pst->stream_xid);
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ continue;
+ }
+ /* Finished processing xact */
+ case 'F':
+ {
+ elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid);
+
+ MemoryContextSwitchTo(ApplyContext);
+
+ CommitTransactionCommand();
+ EndTransactionBlock();
+ CommitTransactionCommand();
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ SpinLockRelease(&pst->mutex);
+
+ continue;
+ }
+ default:
+ break;
+ }
+
+ pq_getmsgint64(&s); // Read LSN info
+ pq_getmsgint64(&s); // TODO Do we need to process it here again somehow?
+ pq_getmsgint64(&s);
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ elog(DEBUG5, "[Apply BGW #%u] applying dispatch for action=%s",
+ pst->n, (char *) &s.data[s.cursor]);
+ apply_dispatch(&s);
+ }
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextReset(ApplyMessageContext);
+ }
+
+ CommitTransactionCommand();
+ EndTransactionBlock();
+ CommitTransactionCommand();
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(ApplyContext);
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ // if (pst->workers_finished == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+
+ elog(LOG, "[Apply BGW #%u] exiting", pst->n);
+
+ /* Signal main process that we are done. */
+ // ConditionVariableBroadcast(&cv);
+ SetLatch(®istrant->procLatch);
+
+ /*
+ * We're done. Explicitly detach the shared memory segment so that we
+ * don't get a resource leak warning at commit time. This will fire any
+ * on_dsm_detach callbacks we've registered, as well. Once that's done,
+ * we can go ahead and exit.
+ */
+ dsm_detach(seg);
+ proc_exit(0);
+}
+
+/*
+ * When we receive a SIGTERM, we set InterruptPending and ProcDiePending just
+ * like a normal backend. The next CHECK_FOR_INTERRUPTS() will do the right
+ * thing.
+ */
+static void
+handle_sigterm(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ SetLatch(MyLatch);
+
+ if (!proc_exit_inprogress)
+ {
+ InterruptPending = true;
+ ProcDiePending = true;
+ }
+
+ errno = save_errno;
+}
+
+/*
+ * Set up a dynamic shared memory segment.
+ *
+ * We set up a control region that contains a ParallelState,
+ * plus one region per message queue. There are as many message queues as
+ * the number of workers.
+ */
+static void
+setup_dsm(WorkerState *wstate)
+{
+ shm_toc_estimator e;
+ int toc_key = 0;
+ Size segsize;
+ dsm_segment *seg;
+ shm_toc *toc;
+ ParallelState *pst;
+ shm_mq *mq;
+ int64 queue_size = 160000000; /* 16 MB for now */
+
+ /* Ensure a valid queue size. */
+ if (queue_size < 0 || ((uint64) queue_size) < shm_mq_minimum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size must be at least %zu bytes",
+ shm_mq_minimum_size)));
+ if (queue_size != ((Size) queue_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size overflows size_t")));
+
+ /*
+ * Estimate how much shared memory we need.
+ *
+ * Because the TOC machinery may choose to insert padding of oddly-sized
+ * requests, we must estimate each chunk separately.
+ *
+ * We need one key to register the location of the header, and we need
+ * nworkers keys to track the locations of the message queues.
+ */
+ shm_toc_initialize_estimator(&e);
+ shm_toc_estimate_chunk(&e, sizeof(ParallelState));
+ shm_toc_estimate_chunk(&e, (Size) queue_size);
+
+ shm_toc_estimate_keys(&e, 1 + 1);
+ segsize = shm_toc_estimate(&e);
+
+ /* Create the shared memory segment and establish a table of contents. */
+ seg = dsm_create(shm_toc_estimate(&e), 0);
+ toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg),
+ segsize);
+
+ /* Set up the header region. */
+ pst = shm_toc_allocate(toc, sizeof(ParallelState));
+ SpinLockInit(&pst->mutex);
+ pst->attached = false;
+ pst->ready = false;
+ pst->finished = false;
+ pst->database_id = MyDatabaseId;
+ pst->subid = MyLogicalRepWorker->subid;
+ pst->stream_xid = stream_xid;
+ pst->authenticated_user_id = GetAuthenticatedUserId();
+ pst->n = nworkers + 1;
+ // ConditionVariableInit(&pst->cv);
+
+ shm_toc_insert(toc, toc_key++, pst);
+
+ /* Set up one message queue per worker, plus one. */
+ mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size),
+ (Size) queue_size);
+ shm_toc_insert(toc, toc_key++, mq);
+ shm_mq_set_sender(mq, MyProc);
+
+ /* Attach the queues. */
+ wstate->mq_handle = shm_mq_attach(mq, seg, wstate->handle);
+
+ /* Return results to caller. */
+ wstate->dsm_seg = seg;
+ wstate->pstate = pst;
+}
+
+/*
+ * Register background workers.
+ */
+static void
+setup_background_worker(WorkerState *wstate)
+{
+ MemoryContext oldcontext;
+ BackgroundWorker worker;
+
+ elog(LOG, "setting up apply worker #%u", nworkers + 1);
+
+ /*
+ * TOCHECK: We need the worker_state object and the background worker handles to
+ * which it points to be allocated in TopMemoryContext rather than
+ * ApplyMessageContext; otherwise, they'll be destroyed before the on_dsm_detach
+ * hooks run.
+ */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+
+ setup_dsm(wstate);
+
+ /*
+ * Arrange to kill all the workers if we abort before all workers are
+ * finished hooking themselves up to the dynamic shared memory segment.
+ *
+ * If we die after all the workers have finished hooking themselves up to
+ * the dynamic shared memory segment, we'll mark the two queues to which
+ * we're directly connected as detached, and the worker(s) connected to
+ * those queues will exit, marking any other queues to which they are
+ * connected as detached. This will cause any as-yet-unaware workers
+ * connected to those queues to exit in their turn, and so on, until
+ * everybody exits.
+ *
+ * But suppose the workers which are supposed to connect to the queues to
+ * which we're directly attached exit due to some error before they
+ * actually attach the queues. The remaining workers will have no way of
+ * knowing this. From their perspective, they're still waiting for those
+ * workers to start, when in fact they've already died.
+ */
+ on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ /* Configure a worker. */
+ MemSet(&worker, 0, sizeof(BackgroundWorker));
+
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_ConsistentState;
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_notify_pid = MyProcPid;
+ sprintf(worker.bgw_library_name, "postgres");
+ sprintf(worker.bgw_function_name, "LogicalApplyBgwMain");
+
+ worker.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(wstate->dsm_seg));
+
+ /* Register the workers. */
+ snprintf(worker.bgw_name, BGW_MAXLEN,
+ "logical replication apply worker #%u for subscription %u",
+ nworkers + 1, MySubscription->oid);
+ if (!RegisterDynamicBackgroundWorker(&worker, &wstate->handle))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not register background process"),
+ errhint("You may need to increase max_worker_processes.")));
+
+ /* All done. */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Wait for worker to become ready. */
+ wait_for_worker(wstate);
+
+ /*
+ * Once we reach this point, all workers are ready. We no longer need to
+ * kill them if we die; they'll die on their own as the message queues
+ * shut down.
+ */
+ cancel_on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ nworkers += 1;
+}
+
+static void
+cleanup_background_worker(dsm_segment *seg, Datum arg)
+{
+ WorkerState *wstate = (WorkerState *) DatumGetPointer(arg);
+
+ TerminateBackgroundWorker(wstate->handle);
+}
+
+static void
+wait_for_worker(WorkerState *wstate)
+{
+ bool result = false;
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool ready;
+
+ /* If the worker is ready, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ ready = wstate->pstate->ready;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (ready)
+ {
+ result = true;
+ break;
+ }
+
+ /* If any workers (or the postmaster) have died, we have failed. */
+ if (!check_worker_status(wstate))
+ {
+ result = false;
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ // ConditionVariableCancelSleep();
+
+ if (!result)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("one or more background workers failed to start")));
+}
+
+static bool
+check_worker_status(WorkerState *wstate)
+{
+ BgwHandleStatus status;
+ pid_t pid;
+
+ status = GetBackgroundWorkerPid(wstate->handle, &pid);
+ if (status == BGWH_STOPPED || status == BGWH_POSTMASTER_DIED)
+ return false;
+
+ /* Otherwise, things still look OK. */
+ return true;
+}
+
+static void
+wait_for_worker_to_finish(WorkerState *wstate)
+{
+ elog(LOG, "waiting for apply worker #%u to finish processing xid %u",
+ wstate->pstate->n, wstate->xid);
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool finished;
+
+ /* If the worker is finished, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ finished = wstate->pstate->finished;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (finished)
+ {
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3a89e23488..7c72db9e83 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -819,6 +819,7 @@ typedef enum
WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATING,
WAIT_EVENT_HASH_GROW_BUCKETS_ELECTING,
WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY,
WAIT_EVENT_LOGICAL_SYNC_DATA,
WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
WAIT_EVENT_MQ_INTERNAL,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 802275311d..afb15c2736 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -122,12 +122,10 @@ extern TransactionId logicalrep_read_stream_stop(StringInfo in);
extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn);
-extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+extern void logicalrep_read_stream_commit(StringInfo out,
LogicalRepCommitData *commit_data);
extern void logicalrep_write_stream_abort(StringInfo out,
TransactionId xid, TransactionId subxid);
-extern void logicalrep_read_stream_abort(StringInfo in,
- TransactionId *xid, TransactionId *subxid);
#endif /* LOGICALREP_PROTO_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index e9524aefd9..30ad40247d 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -13,6 +13,7 @@
#define LOGICALWORKER_H
extern void ApplyWorkerMain(Datum main_arg);
+extern void LogicalApplyBgwMain(Datum main_arg);
extern bool IsLogicalWorker(void);
--
2.17.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-08-28 19:06 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-08-28 19:06 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
On Wed, Aug 28, 2019 at 08:17:47PM +0300, Alexey Kondratov wrote:
>Hi Tomas,
>
>>>>>Interesting. Any idea where does the extra overhead in this particular
>>>>>case come from? It's hard to deduce that from the single flame graph,
>>>>>when I don't have anything to compare it with (i.e. the flame
>>>>>graph for
>>>>>the "normal" case).
>>>>I guess that bottleneck is in disk operations. You can check
>>>>logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
>>>>writes (~26%) take around 35% of CPU time in summary. To compare,
>>>>please, see attached flame graph for the following transaction:
>>>>
>>>>INSERT INTO large_text
>>>>SELECT (SELECT string_agg('x', ',')
>>>>FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>>>>
>>>>Execution Time: 44519.816 ms
>>>>Time: 98333,642 ms (01:38,334)
>>>>
>>>>where disk IO is only ~7-8% in total. So we get very roughly the same
>>>>~x4-5 performance drop here. JFYI, I am using a machine with SSD
>>>>for tests.
>>>>
>>>>Therefore, probably you may write changes on receiver in bigger chunks,
>>>>not each change separately.
>>>>
>>>Possibly, I/O is certainly a possible culprit, although we should be
>>>using buffered I/O and there certainly are not any fsyncs here. So I'm
>>>not sure why would it be cheaper to do the writes in batches.
>>>
>>>BTW does this mean you see the overhead on the apply side? Or are you
>>>running this on a single machine, and it's difficult to decide?
>>
>>I run this on a single machine, but walsender and worker are
>>utilizing almost 100% of CPU per each process all the time, and at
>>apply side I/O syscalls take about 1/3 of CPU time. Though I am
>>still not sure, but for me this result somehow links performance
>>drop with problems at receiver side.
>>
>>Writing in batches was just a hypothesis and to validate it I have
>>performed test with large txn, but consisting of a smaller number of
>>wide rows. This test does not exhibit any significant performance
>>drop, while it was streamed too. So it seems to be valid. Anyway, I
>>do not have other reasonable ideas beside that right now.
>
>I've checked recently this patch again and tried to elaborate it in
>terms of performance. As a result I've implemented a new POC version
>of the applier (attached). Almost everything in streaming logic stayed
>intact, but apply worker is significantly different.
>
>As I wrote earlier I still claim, that spilling changes on disk at the
>applier side adds additional overhead, but it is possible to get rid
>of it. In my additional patch I do the following:
>
>1) Maintain a pool of additional background workers (bgworkers), that
>are connected with main logical apply worker via shm_mq's. Each worker
>is dedicated to the processing of specific streamed transaction.
>
>2) When we receive a streamed change for some transaction, we check
>whether there is an existing dedicated bgworker in HTAB (xid ->
>bgworker), or there are some in the idle list, or spawn a new one.
>
>3) We pass all changes (between STREAM START/STOP) to that bgworker
>via shm_mq_send without intermediate waiting. However, we wait for
>bgworker to apply the entire changes chunk at STREAM STOP, since we
>don't want transactions reordering.
>
>4) When transaction is commited/aborted worker is being added to the
>idle list and is waiting for reassigning message.
>
>5) I have used the same machinery with apply_dispatch in bgworkers,
>since most of actions are practically very similar.
>
>Thus, we do not spill anything at the applier side, so transaction
>changes are processed by bgworkers as normal backends do. In the same
>time, changes processing is strictly serial, which prevents
>transactions reordering and possible conflicts/anomalies. Even though
>we trade off performance in favor of stability the result is rather
>impressive. I have used a similar query for testing as before:
>
>EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1, num2, num3)
> SELECT round(random()*10), random(), random()*142
> FROM generate_series(1, 1000000) s(i);
>
>with 1kk (1000000), 3kk and 5kk rows; logical_work_mem = 64MB and
>synchronous_standby_names = 'FIRST 1 (large_sub)'. Table schema is
>following:
>
>CREATE TABLE large_test (
> id serial primary key,
> num1 bigint,
> num2 double precision,
> num3 double precision
>);
>
>Here are the results:
>
>-------------------------------------------------------------------
>| N | Time on master, sec | Total xact time, sec | Ratio |
>-------------------------------------------------------------------
>| On commit (master, v13) |
>-------------------------------------------------------------------
>| 1kk | 6.5 | 17.6 | x2.74 |
>-------------------------------------------------------------------
>| 3kk | 21 | 55.4 | x2.64 |
>-------------------------------------------------------------------
>| 5kk | 38.3 | 91.5 | x2.39 |
>-------------------------------------------------------------------
>| Stream + spill |
>-------------------------------------------------------------------
>| 1kk | 5.9 | 18 | x3 |
>-------------------------------------------------------------------
>| 3kk | 19.5 | 52.4 | x2.7 |
>-------------------------------------------------------------------
>| 5kk | 33.3 | 86.7 | x2.86 |
>-------------------------------------------------------------------
>| Stream + BGW pool |
>-------------------------------------------------------------------
>| 1kk | 6 | 12 | x2 |
>-------------------------------------------------------------------
>| 3kk | 18.5 | 30.5 | x1.65 |
>-------------------------------------------------------------------
>| 5kk | 35.6 | 53.9 | x1.51 |
>-------------------------------------------------------------------
>
>It seems that overhead added by synchronous replica is lower by 2-3
>times compared with Postgres master and streaming with spilling.
>Therefore, the original patch eliminated delay before large
>transaction processing start by sender, while this additional patch
>speeds up the applier side.
>
>Although the overall speed up is surely measurable, there is a room
>for improvements yet:
>
>1) Currently bgworkers are only spawned on demand without some initial
>pool and never stopped. Maybe we should create a small pool on
>replication start and offload some of idle bgworkers if they exceed
>some limit?
>
>2) Probably we can track somehow that incoming change has conflicts
>with some of being processed xacts, so we can wait for specific
>bgworkers only in that case?
>
>3) Since the communication between main logical apply worker and each
>bgworker from the pool is a 'single producer --- single consumer'
>problem, then probably it is possible to wait and set/check flags
>without locks, but using just atomics.
>
>What do you think about this concept in general? Any concerns and
>criticism are welcome!
>
Hi Alexey,
I'm unable to do any in-depth review of the patch over the next two weeks
or so, but I think the idea of having a pool of apply workers is sound and
can be quite beneficial for some workloads.
I don't think it matters very much whether the workers are started at the
beginning or allocated ad hoc, that's IMO a minor implementation detail.
There's one huge challenge that I however don't see mentioned in your
message or in the patch (after cursory reading) - ensuring the same commit
order, and introducing deadlocks that would not exist in single-process
apply.
Surely, we want to end up with the same commit order as on the upstream,
otherwise we might easily get different data on the subscriber. So when we
pass the large transaction to a separate process, then this process has
to wait for the other processes processing transactions that committed
first. And similarly, other processes have to wait for this process.
Depending on the commit order. I might have missed something, but I don't
see anything like that in your patch.
Essentially, this means there needs to be some sort of wait between those
apply processes, enforcing the commit order.
That however means we can easily introduce deadlocks into workloads where
the serial-apply would not have that issue - imagine multiple large
transactions, touching the same set of rows. We may ship them to different
bgworkers, and those processes may deadlock.
Of course, the deadlock detector will come around (assuming the wait is
done in a way visible to the detector) and will abort one of the
processes. But we don't know it'll abort the right one - it may easily
abort the apply process that needs to comit first, and eveyone else is
waitiing for it. Which stalls the apply forever.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-08-29 14:37 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-08-29 14:37 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
On 28.08.2019 22:06, Tomas Vondra wrote:
>
>>
>>>>>> Interesting. Any idea where does the extra overhead in this
>>>>>> particular
>>>>>> case come from? It's hard to deduce that from the single flame
>>>>>> graph,
>>>>>> when I don't have anything to compare it with (i.e. the flame
>>>>>> graph for
>>>>>> the "normal" case).
>>>>> I guess that bottleneck is in disk operations. You can check
>>>>> logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
>>>>> writes (~26%) take around 35% of CPU time in summary. To compare,
>>>>> please, see attached flame graph for the following transaction:
>>>>>
>>>>> INSERT INTO large_text
>>>>> SELECT (SELECT string_agg('x', ',')
>>>>> FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>>>>>
>>>>> Execution Time: 44519.816 ms
>>>>> Time: 98333,642 ms (01:38,334)
>>>>>
>>>>> where disk IO is only ~7-8% in total. So we get very roughly the same
>>>>> ~x4-5 performance drop here. JFYI, I am using a machine with SSD
>>>>> for tests.
>>>>>
>>>>> Therefore, probably you may write changes on receiver in bigger
>>>>> chunks,
>>>>> not each change separately.
>>>>>
>>>> Possibly, I/O is certainly a possible culprit, although we should be
>>>> using buffered I/O and there certainly are not any fsyncs here. So I'm
>>>> not sure why would it be cheaper to do the writes in batches.
>>>>
>>>> BTW does this mean you see the overhead on the apply side? Or are you
>>>> running this on a single machine, and it's difficult to decide?
>>>
>>> I run this on a single machine, but walsender and worker are
>>> utilizing almost 100% of CPU per each process all the time, and at
>>> apply side I/O syscalls take about 1/3 of CPU time. Though I am
>>> still not sure, but for me this result somehow links performance
>>> drop with problems at receiver side.
>>>
>>> Writing in batches was just a hypothesis and to validate it I have
>>> performed test with large txn, but consisting of a smaller number of
>>> wide rows. This test does not exhibit any significant performance
>>> drop, while it was streamed too. So it seems to be valid. Anyway, I
>>> do not have other reasonable ideas beside that right now.
>>
>> It seems that overhead added by synchronous replica is lower by 2-3
>> times compared with Postgres master and streaming with spilling.
>> Therefore, the original patch eliminated delay before large
>> transaction processing start by sender, while this additional patch
>> speeds up the applier side.
>>
>> Although the overall speed up is surely measurable, there is a room
>> for improvements yet:
>>
>> 1) Currently bgworkers are only spawned on demand without some
>> initial pool and never stopped. Maybe we should create a small pool
>> on replication start and offload some of idle bgworkers if they
>> exceed some limit?
>>
>> 2) Probably we can track somehow that incoming change has conflicts
>> with some of being processed xacts, so we can wait for specific
>> bgworkers only in that case?
>>
>> 3) Since the communication between main logical apply worker and each
>> bgworker from the pool is a 'single producer --- single consumer'
>> problem, then probably it is possible to wait and set/check flags
>> without locks, but using just atomics.
>>
>> What do you think about this concept in general? Any concerns and
>> criticism are welcome!
>>
>
Hi Tomas,
Thank you for a quick response.
> I don't think it matters very much whether the workers are started at the
> beginning or allocated ad hoc, that's IMO a minor implementation detail.
OK, I had the same vision about this point. Any minor differences here
will be neglectable for a sufficiently large transaction.
>
> There's one huge challenge that I however don't see mentioned in your
> message or in the patch (after cursory reading) - ensuring the same
> commit
> order, and introducing deadlocks that would not exist in single-process
> apply.
Probably I haven't explained well this part, sorry for that. In my patch
I don't use workers pool for a concurrent transaction apply, but rather
for a fast context switch between long-lived streamed transactions. In
other words we apply all changes arrived from the sender in a completely
serial manner. Being written step-by-step it looks like:
1) Read STREAM START message and figure out the target worker by xid.
2) Put all changes, which belongs to this xact to the selected worker
one by one via shm_mq_send.
3) Read STREAM STOP message and wait until our worker will apply all
changes in the queue.
4) Process all other chunks of streamed xacts in the same manner.
5) Process all non-streamed xacts immediately in the main apply worker loop.
6) If we read STREAMED COMMIT/ABORT we again wait until selected worker
either commits or aborts.
Thus, it automatically guaranties the same commit order on replica as on
master. Yes, we loose some performance here, since we don't apply
transactions concurrently, but it would bring all those problems you
have described.
However, you helped me to figure out another point I have forgotten.
Although we ensure commit order automatically, the beginning of streamed
xacts may reorder. It happens if some small xacts have been commited on
master since the streamed one started, because we do not start streaming
immediately, but only after logical_work_mem hit. I have performed some
tests with conflicting xacts and it seems that it's not a problem, since
locking mechanism in Postgres guarantees that if there would some
deadlocks, they will happen earlier on master. So if some records hit
the WAL, it is safe to apply the sequentially. Am I wrong?
Anyway, I'm going to double check the safety of this part later.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-08-29 18:48 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-08-29 18:48 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
On Thu, Aug 29, 2019 at 05:37:45PM +0300, Alexey Kondratov wrote:
>On 28.08.2019 22:06, Tomas Vondra wrote:
>>
>>>
>>>>>>>Interesting. Any idea where does the extra overhead in
>>>>>>>this particular
>>>>>>>case come from? It's hard to deduce that from the single
>>>>>>>flame graph,
>>>>>>>when I don't have anything to compare it with (i.e. the
>>>>>>>flame graph for
>>>>>>>the "normal" case).
>>>>>>I guess that bottleneck is in disk operations. You can check
>>>>>>logical_repl_worker_new_perf.svg flame graph: disk reads (~9%) and
>>>>>>writes (~26%) take around 35% of CPU time in summary. To compare,
>>>>>>please, see attached flame graph for the following transaction:
>>>>>>
>>>>>>INSERT INTO large_text
>>>>>>SELECT (SELECT string_agg('x', ',')
>>>>>>FROM generate_series(1, 2000)) FROM generate_series(1, 1000000);
>>>>>>
>>>>>>Execution Time: 44519.816 ms
>>>>>>Time: 98333,642 ms (01:38,334)
>>>>>>
>>>>>>where disk IO is only ~7-8% in total. So we get very roughly the same
>>>>>>~x4-5 performance drop here. JFYI, I am using a machine with
>>>>>>SSD for tests.
>>>>>>
>>>>>>Therefore, probably you may write changes on receiver in
>>>>>>bigger chunks,
>>>>>>not each change separately.
>>>>>>
>>>>>Possibly, I/O is certainly a possible culprit, although we should be
>>>>>using buffered I/O and there certainly are not any fsyncs here. So I'm
>>>>>not sure why would it be cheaper to do the writes in batches.
>>>>>
>>>>>BTW does this mean you see the overhead on the apply side? Or are you
>>>>>running this on a single machine, and it's difficult to decide?
>>>>
>>>>I run this on a single machine, but walsender and worker are
>>>>utilizing almost 100% of CPU per each process all the time, and
>>>>at apply side I/O syscalls take about 1/3 of CPU time. Though I
>>>>am still not sure, but for me this result somehow links
>>>>performance drop with problems at receiver side.
>>>>
>>>>Writing in batches was just a hypothesis and to validate it I
>>>>have performed test with large txn, but consisting of a smaller
>>>>number of wide rows. This test does not exhibit any significant
>>>>performance drop, while it was streamed too. So it seems to be
>>>>valid. Anyway, I do not have other reasonable ideas beside that
>>>>right now.
>>>
>>>It seems that overhead added by synchronous replica is lower by
>>>2-3 times compared with Postgres master and streaming with
>>>spilling. Therefore, the original patch eliminated delay before
>>>large transaction processing start by sender, while this
>>>additional patch speeds up the applier side.
>>>
>>>Although the overall speed up is surely measurable, there is a
>>>room for improvements yet:
>>>
>>>1) Currently bgworkers are only spawned on demand without some
>>>initial pool and never stopped. Maybe we should create a small
>>>pool on replication start and offload some of idle bgworkers if
>>>they exceed some limit?
>>>
>>>2) Probably we can track somehow that incoming change has
>>>conflicts with some of being processed xacts, so we can wait for
>>>specific bgworkers only in that case?
>>>
>>>3) Since the communication between main logical apply worker and
>>>each bgworker from the pool is a 'single producer --- single
>>>consumer' problem, then probably it is possible to wait and
>>>set/check flags without locks, but using just atomics.
>>>
>>>What do you think about this concept in general? Any concerns and
>>>criticism are welcome!
>>>
>>
>
>Hi Tomas,
>
>Thank you for a quick response.
>
>>I don't think it matters very much whether the workers are started at the
>>beginning or allocated ad hoc, that's IMO a minor implementation detail.
>
>OK, I had the same vision about this point. Any minor differences here
>will be neglectable for a sufficiently large transaction.
>
>>
>>There's one huge challenge that I however don't see mentioned in your
>>message or in the patch (after cursory reading) - ensuring the same
>>commit
>>order, and introducing deadlocks that would not exist in single-process
>>apply.
>
>Probably I haven't explained well this part, sorry for that. In my
>patch I don't use workers pool for a concurrent transaction apply, but
>rather for a fast context switch between long-lived streamed
>transactions. In other words we apply all changes arrived from the
>sender in a completely serial manner. Being written step-by-step it
>looks like:
>
>1) Read STREAM START message and figure out the target worker by xid.
>
>2) Put all changes, which belongs to this xact to the selected worker
>one by one via shm_mq_send.
>
>3) Read STREAM STOP message and wait until our worker will apply all
>changes in the queue.
>
>4) Process all other chunks of streamed xacts in the same manner.
>
>5) Process all non-streamed xacts immediately in the main apply worker loop.
>
>6) If we read STREAMED COMMIT/ABORT we again wait until selected
>worker either commits or aborts.
>
>Thus, it automatically guaranties the same commit order on replica as
>on master. Yes, we loose some performance here, since we don't apply
>transactions concurrently, but it would bring all those problems you
>have described.
>
OK, so it's apply in multiple processes, but at any moment only a single
apply process is active.
>However, you helped me to figure out another point I have forgotten.
>Although we ensure commit order automatically, the beginning of
>streamed xacts may reorder. It happens if some small xacts have been
>commited on master since the streamed one started, because we do not
>start streaming immediately, but only after logical_work_mem hit. I
>have performed some tests with conflicting xacts and it seems that
>it's not a problem, since locking mechanism in Postgres guarantees
>that if there would some deadlocks, they will happen earlier on
>master. So if some records hit the WAL, it is safe to apply the
>sequentially. Am I wrong?
>
I think you're right the way you interleave the changes ensures you
can't introduce new deadlocks between transactions in this stream. I don't
think reordering the blocks of streamed trasactions does matter, as long
as the comit order is ensured in this case.
>Anyway, I'm going to double check the safety of this part later.
>
OK.
FWIW my understanding is that the speedup comes mostly from elimination of
the serialization to a file. That however requires savepoints to handle
aborts of subtransactions - I'm pretty sure I'd be trivial to create a
workload where this will be much slower (with many aborts of large
subtransactions).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-08-30 15:59 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Konstantin Knizhnik @ 2019-08-30 15:59 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
>
> FWIW my understanding is that the speedup comes mostly from
> elimination of
> the serialization to a file. That however requires savepoints to handle
> aborts of subtransactions - I'm pretty sure I'd be trivial to create a
> workload where this will be much slower (with many aborts of large
> subtransactions).
>
>
I think that instead of defining savepoints it is simpler and more
efficient to use
BeginInternalSubTransaction +
ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction
as it is done in PL/pgSQL (pl_exec.c).
Not sure if it can pr
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-02 22:06 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 2 replies; 536+ messages in thread
From: Alvaro Herrera @ 2019-09-02 22:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
In the interest of moving things forward, how far are we from making
0001 committable? If I understand correctly, the rest of this patchset
depends on https://commitfest.postgresql.org/24/944/ which seems to be
moving at a glacial pace (or, actually, slower, because glaciers do
move, which cannot be said of that other patch.)
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-03 10:39 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-03 10:39 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: pgsql-hackers
On Mon, Sep 02, 2019 at 06:06:50PM -0400, Alvaro Herrera wrote:
>In the interest of moving things forward, how far are we from making
>0001 committable? If I understand correctly, the rest of this patchset
>depends on https://commitfest.postgresql.org/24/944/ which seems to be
>moving at a glacial pace (or, actually, slower, because glaciers do
>move, which cannot be said of that other patch.)
>
I think 0001 is mostly there. I think there's one bug in this patch
version, but I need to check and I'll post an updated version shortly if
needed.
FWIW maybe we should stop comparing things to glaciers. 50 years from not
people won't know what a glacier is, and it'll be just like the floppy
icon on the save button.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-16 16:54 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-09-16 16:54 UTC (permalink / raw)
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
>>
>> FWIW my understanding is that the speedup comes mostly from
>> elimination of
>> the serialization to a file. That however requires savepoints to handle
>> aborts of subtransactions - I'm pretty sure I'd be trivial to create a
>> workload where this will be much slower (with many aborts of large
>> subtransactions).
>>
Yes, and it was my main motivation to eliminate that extra serialization
to file. I've experimented a bit with large transactions + savepoints +
aborts and ended up with a following query (the same schema as before
with 600k rows):
BEGIN;
SAVEPOINT s1;
UPDATE large_test SET num1 = num1 + 1, num2 = num2 + 1, num3 = num3 + 1;
SAVEPOINT s2;
UPDATE large_test SET num1 = num1 + 1, num2 = num2 + 1, num3 = num3 + 1;
SAVEPOINT s3;
UPDATE large_test SET num1 = num1 + 1, num2 = num2 + 1, num3 = num3 + 1;
ROLLBACK TO SAVEPOINT s3;
ROLLBACK TO SAVEPOINT s2;
ROLLBACK TO SAVEPOINT s1;
END;
It looks like the worst case scenario, as we do a lot of work and then
abort all subxacts one by one. As expected,it takes much longer (up to
x30) to process using background worker instead of spilling to file.
Surely, it is much easier to truncate a file, than apply all changes +
abort. However, I guess that this kind of load pattern is not the most
typical for real-life applications.
Also this test helped me to find a bug in my current savepoints routine,
so new patch is attached.
On 30.08.2019 18:59, Konstantin Knizhnik wrote:
>
> I think that instead of defining savepoints it is simpler and more
> efficient to use
>
> BeginInternalSubTransaction +
> ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction
>
> as it is done in PL/pgSQL (pl_exec.c).
> Not sure if it can pr
>
Both BeginInternalSubTransaction and DefineSavepoint use
PushTransaction() internally for a normal subtransaction start. So they
seems to be identical from the performance perspective, which is also
stated in the comment section:
/*
* BeginInternalSubTransaction
* This is the same as DefineSavepoint except it allows
TBLOCK_STARTED,
* TBLOCK_IMPLICIT_INPROGRESS, TBLOCK_END, and TBLOCK_PREPARE
states,
* and therefore it can safely be used in functions that might
be called
* when not inside a BEGIN block or when running deferred
triggers at
* COMMIT/PREPARE time. Also, it automatically does
* CommitTransactionCommand/StartTransactionCommand instead of
expecting
* the caller to do it.
*/
Please, correct me if I'm wrong.
Anyway, I've performed a profiling of my apply worker (flamegraph is
attached) and it spends the vast amount of time (>90%) applying changes.
So the problem is not in the savepoints their-self, but in the fact that
we first apply all changes and then abort all the work. Not sure, that
it is possible to do something in this case.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
Attachments:
worker_aborts_perf.zip (application/zip, 69.8K) download
v2-0011-BGWorkers-pool-for-streamed-transactions-apply-wi.patch (text/x-patch, 60.1K) download
From 7ffeeb69fd626dbcc35ca13d97a8407573ea6d4a Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <kondratov.aleksey@gmail.com>
Date: Wed, 28 Aug 2019 15:26:50 +0300
Subject: [PATCH v2 11/11] BGWorkers pool for streamed transactions apply
without spilling on disk
---
src/backend/postmaster/bgworker.c | 3 +
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/logical/proto.c | 17 +-
src/backend/replication/logical/worker.c | 1783 +++++++++++-----------
src/include/pgstat.h | 1 +
src/include/replication/logicalproto.h | 4 +-
src/include/replication/logicalworker.h | 1 +
7 files changed, 936 insertions(+), 876 deletions(-)
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index f5db5a8c4a..6860df07ca 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "LogicalApplyBgwMain", LogicalApplyBgwMain
}
};
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index e5a4d147a7..b32994784f 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3637,6 +3637,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING:
event_name = "Hash/GrowBuckets/Reinserting";
break;
+ case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY:
+ event_name = "LogicalApplyWorkerReady";
+ break;
case WAIT_EVENT_LOGICAL_SYNC_DATA:
event_name = "LogicalSyncData";
break;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 4bec9fe8b5..954ce7343a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -789,14 +789,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
pq_sendint64(out, txn->commit_time);
}
-TransactionId
+void
logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
{
- TransactionId xid;
uint8 flags;
- xid = pq_getmsgint(in, 4);
-
/* read flags (unused for now) */
flags = pq_getmsgbyte(in);
@@ -807,8 +804,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
commit_data->commit_lsn = pq_getmsgint64(in);
commit_data->end_lsn = pq_getmsgint64(in);
commit_data->committime = pq_getmsgint64(in);
-
- return xid;
}
void
@@ -823,13 +818,3 @@ logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
pq_sendint32(out, xid);
pq_sendint32(out, subxid);
}
-
-void
-logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid)
-{
- Assert(xid && subxid);
-
- *xid = pq_getmsgint(in, 4);
- *subxid = pq_getmsgint(in, 4);
-}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ca632b7dc4..ab43b12985 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -92,11 +92,16 @@
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+// #include "storage/condition_variable.h"
+#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/shm_mq.h"
+#include "storage/shm_toc.h"
+#include "storage/spin.h"
#include "tcop/tcopprot.h"
@@ -115,6 +120,54 @@
#include "utils/syscache.h"
#define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */
+#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change
+
+typedef struct ParallelState
+{
+ slock_t mutex;
+ // ConditionVariable cv;
+ bool attached;
+ bool ready;
+ bool finished;
+ Oid database_id;
+ Oid authenticated_user_id;
+ Oid subid;
+ Oid stream_xid;
+ uint32 n;
+} ParallelState;
+
+typedef struct WorkerState
+{
+ TransactionId xid;
+ BackgroundWorkerHandle *handle;
+ shm_mq_handle *mq_handle;
+ dsm_segment *dsm_seg;
+ ParallelState volatile *pstate;
+} WorkerState;
+
+/* Apply workers hash table (initialized on first use) */
+static HTAB *ApplyWorkersHash = NULL;
+static WorkerState **ApplyWorkersIdleList = NULL;
+static uint32 pool_size = 10; /* MaxConnections default? */
+static uint32 nworkers = 0;
+static uint32 nfreeworkers = 0;
+
+/* Fields valid only for apply background workers */
+bool isLogicalApplyWorker = false;
+volatile ParallelState *MyParallelState = NULL;
+
+/* Worker setup and interactions */
+static void setup_dsm(WorkerState *wstate);
+static void setup_background_worker(WorkerState *wstate);
+static void cleanup_background_worker(dsm_segment *seg, Datum arg);
+static void handle_sigterm(SIGNAL_ARGS);
+
+static bool check_worker_status(WorkerState *wstate);
+static void wait_for_worker(WorkerState *wstate);
+static void wait_for_worker_to_finish(WorkerState *wstate);
+
+static WorkerState * find_or_start_worker(TransactionId xid, bool start);
+static void stop_worker(WorkerState *wstate);
typedef struct FlushPosition
{
@@ -143,47 +196,13 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
-/* fields valid only when processing streamed transaction */
+/* Fields valid only when processing streamed transaction */
bool in_streamed_transaction = false;
static TransactionId stream_xid = InvalidTransactionId;
-static int stream_fd = -1;
-
-typedef struct SubXactInfo
-{
- TransactionId xid; /* XID of the subxact */
- off_t offset; /* offset in the file */
-} SubXactInfo;
-
-static uint32 nsubxacts = 0;
-static uint32 nsubxacts_max = 0;
-static SubXactInfo * subxacts = NULL;
-static TransactionId subxact_last = InvalidTransactionId;
-
-static void subxact_filename(char *path, Oid subid, TransactionId xid);
-static void changes_filename(char *path, Oid subid, TransactionId xid);
-
-/*
- * Information about subtransactions of a given toplevel transaction.
- */
-static void subxact_info_write(Oid subid, TransactionId xid);
-static void subxact_info_read(Oid subid, TransactionId xid);
-static void subxact_info_add(TransactionId xid);
-
-/*
- * Serialize and deserialize changes for a toplevel transaction.
- */
-static void stream_cleanup_files(Oid subid, TransactionId xid);
-static void stream_open_file(Oid subid, TransactionId xid, bool first);
-static void stream_write_change(char action, StringInfo s);
-static void stream_close_file(void);
-
-/*
- * Array of serialized XIDs.
- */
-static int nxids = 0;
-static int maxnxids = 0;
-static TransactionId *xids = NULL;
+static TransactionId current_xid = InvalidTransactionId;
+static TransactionId prev_xid = InvalidTransactionId;
+static uint32 nchanges = 0;
static bool handle_streamed_transaction(const char action, StringInfo s);
@@ -199,6 +218,16 @@ static volatile sig_atomic_t got_SIGHUP = false;
/* prototype needed because of stream_commit */
static void apply_dispatch(StringInfo s);
+// /* Debug only */
+// static void
+// iter_sleep(int seconds)
+// {
+// for (int i = 0; i < seconds; i++)
+// {
+// pg_usleep(1 * 1000L * 1000L);
+// }
+// }
+
/*
* Should this worker apply changes for given relation.
*
@@ -250,6 +279,107 @@ ensure_transaction(void)
return true;
}
+/*
+ * Look up worker inside ApplyWorkersHash for requested xid.
+ * Throw error if not found or start a new one if start=true is passed.
+ */
+static WorkerState *
+find_or_start_worker(TransactionId xid, bool start)
+{
+ bool found;
+ WorkerState *entry = NULL;
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* First time through, initialize apply workers hashtable */
+ if (ApplyWorkersHash == NULL)
+ {
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(WorkerState);
+ ctl.hcxt = ApplyContext; /* Allocate ApplyWorkersHash in the ApplyContext */
+ ApplyWorkersHash = hash_create("logical apply workers hash", 8,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ }
+
+ Assert(ApplyWorkersHash != NULL);
+
+ /*
+ * Find entry for requested transaction.
+ */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found);
+
+ if (!found && start)
+ {
+ /* If there is at least one worker in the idle list, then take one. */
+ if (nfreeworkers > 0)
+ {
+ char action = 'R';
+
+ Assert(ApplyWorkersIdleList != NULL);
+
+ entry = ApplyWorkersIdleList[nfreeworkers - 1];
+ if (!hash_update_hash_key(ApplyWorkersHash,
+ (void *) entry,
+ (void *) &xid))
+ elog(ERROR, "could not reassign apply worker #%u entry from xid %u to xid %u",
+ entry->pstate->n, entry->xid, xid);
+
+ entry->xid = xid;
+ entry->pstate->finished = false;
+ entry->pstate->stream_xid = xid;
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+
+ ApplyWorkersIdleList[--nfreeworkers] = NULL;
+ }
+ else
+ {
+ /* No entry in hash and no idle workers. Create a new one. */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found);
+ entry->xid = xid;
+ setup_background_worker(entry);
+
+ if (nworkers == pool_size)
+ {
+ ApplyWorkersIdleList = repalloc(ApplyWorkersIdleList, pool_size + 10);
+ pool_size += 10;
+ }
+ }
+ }
+ else if (!found && !start)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not find logical apply worker for xid %u", xid)));
+ else
+ elog(DEBUG5, "there is an existing logical apply worker for xid %u", xid);
+
+ Assert(entry != NULL);
+
+ return entry;
+}
+
+/*
+ * Gracefully teardown apply worker.
+ */
+static void
+stop_worker(WorkerState *wstate)
+{
+ /*
+ * Sending zero-length data to worker in order to stop it.
+ */
+ shm_mq_send(wstate->mq_handle, 0, NULL, false);
+
+ elog(LOG, "detaching DSM of apply worker #%u for xid %u",
+ wstate->pstate->n, wstate->xid);
+ dsm_detach(wstate->dsm_seg);
+
+ /* Delete worker entry */
+ (void) hash_search(ApplyWorkersHash, &wstate->xid, HASH_REMOVE, NULL);
+}
+
/*
* Handle streamed transactions.
*
@@ -262,12 +392,12 @@ static bool
handle_streamed_transaction(const char action, StringInfo s)
{
TransactionId xid;
+ WorkerState *entry;
/* not in streaming mode */
- if (!in_streamed_transaction)
+ if (!in_streamed_transaction || isLogicalApplyWorker)
return false;
- Assert(stream_fd != -1);
Assert(TransactionIdIsValid(stream_xid));
/*
@@ -278,11 +408,16 @@ handle_streamed_transaction(const char action, StringInfo s)
Assert(TransactionIdIsValid(xid));
- /* Add the new subxact to the array (unless already there). */
- subxact_info_add(xid);
+ /*
+ * Find worker for requested xid.
+ */
+ entry = find_or_start_worker(stream_xid, false);
- /* write the change to the current file */
- stream_write_change(action, s);
+ // elog(LOG, "sending message of length=%d and raw=%s, action=%s", s->len, s->data, (char *) &action);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
+ nchanges += 1;
+
+ // iter_sleep(3600);
return true;
}
@@ -643,7 +778,8 @@ apply_handle_origin(StringInfo s)
static void
apply_handle_stream_start(StringInfo s)
{
- bool first_segment;
+ bool first_segment;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
@@ -652,17 +788,16 @@ apply_handle_stream_start(StringInfo s)
/* extract XID of the top-level transaction */
stream_xid = logicalrep_read_stream_start(s, &first_segment);
+ nchanges = 0;
- /* open the spool file for this transaction */
- stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, true);
- /*
- * if this is not the first segment, open existing file
- *
- * XXX Note that the cleanup is performed by stream_open_file.
- */
- if (!first_segment)
- subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+ SpinLockAcquire(&entry->pstate->mutex);
+ entry->pstate->ready = false;
+ SpinLockRelease(&entry->pstate->mutex);
+
+ elog(LOG, "starting streaming of xid %u", stream_xid);
pgstat_report_activity(STATE_RUNNING, NULL);
}
@@ -673,16 +808,19 @@ apply_handle_stream_start(StringInfo s)
static void
apply_handle_stream_stop(StringInfo s)
{
+ WorkerState *entry;
+ char action = 'E';
+
Assert(in_streamed_transaction);
- /*
- * Close the file with serialized changes, and serialize information about
- * subxacts for the toplevel transaction.
- */
- subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
- stream_close_file();
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
+
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ wait_for_worker(entry);
in_streamed_transaction = false;
+ elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges);
pgstat_report_activity(STATE_IDLE, NULL);
}
@@ -695,96 +833,67 @@ apply_handle_stream_abort(StringInfo s)
{
TransactionId xid;
TransactionId subxid;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
- logicalrep_read_stream_abort(s, &xid, &subxid);
-
- /*
- * If the two XIDs are the same, it's in fact abort of toplevel xact, so
- * just delete the files with serialized info.
- */
- if (xid == subxid)
+ if(isLogicalApplyWorker)
{
- char path[MAXPGPATH];
+ subxid = pq_getmsgint(s, 4);
- /*
- * XXX Maybe this should be an error instead? Can we receive abort for
- * a toplevel transaction we haven't received?
- */
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u",
+ MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId())));
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ if (subxid == stream_xid)
+ AbortCurrentTransaction();
+ else
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", subxid);
- if (unlink(path) < 0)
- ereport(ERROR,
+ ereport(LOG,
(errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname)));
- subxact_filename(path, MyLogicalRepWorker->subid, xid);
-
- if (unlink(path) < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ RollbackToSavepoint(spname);
+ CommitTransactionCommand();
+ // RollbackAndReleaseCurrentSubTransaction();
- return;
+ pfree(spname);
+ }
}
else
{
- /*
- * OK, so it's a subxact. We need to read the subxact file for the
- * toplevel transaction, determine the offset tracked for the subxact,
- * and truncate the file with changes. We also remove the subxacts
- * with higher offsets (or rather higher XIDs).
- *
- * We intentionally scan the array from the tail, because we're likely
- * aborting a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- *
- * XXX Or perhaps we can rely on the aborts to arrive in the reverse
- * order, i.e. from the inner-most subxact (when nested)? In which
- * case we could simply check the last element.
- */
+ xid = pq_getmsgint(s, 4);
+ subxid = pq_getmsgint(s, 4);
- int64 i;
- int64 subidx;
- bool found = false;
- char path[MAXPGPATH];
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
- subidx = -1;
- subxact_info_read(MyLogicalRepWorker->subid, xid);
+ elog(LOG, "processing abort request of streamed transaction xid %u, subxid %u",
+ xid, subxid);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- /* FIXME optimize the search by bsearch on sorted data */
- for (i = nsubxacts; i > 0; i--)
+ if (subxid == stream_xid)
{
- if (subxacts[i - 1].xid == subxid)
- {
- subidx = (i - 1);
- found = true;
- break;
- }
- }
-
- /* We should not receive aborts for unknown subtransactions. */
- Assert(found);
+ char action = 'F';
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ // shm_mq_send(entry->mq_handle, 0, NULL, false);
- /* OK, truncate the file at the right offset. */
- Assert((subidx >= 0) && (subidx < nsubxacts));
+ wait_for_worker_to_finish(entry);
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
- if (truncate(path, subxacts[subidx].offset))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not truncate file \"%s\": %m", path)));
+ // elog(LOG, "detaching DSM of apply worker for xid=%u\n", entry->xid);
+ // dsm_detach(entry->dsm_seg);
- /* discard the subxacts added later */
- nsubxacts = subidx;
-
- /* write the updated subxact list */
- subxact_info_write(MyLogicalRepWorker->subid, xid);
+ // /* Delete worker entry */
+ // (void) hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL);
+ }
}
}
@@ -794,159 +903,56 @@ apply_handle_stream_abort(StringInfo s)
static void
apply_handle_stream_commit(StringInfo s)
{
- int fd;
TransactionId xid;
- StringInfoData s2;
- int nchanges;
-
- char path[MAXPGPATH];
- char *buffer = NULL;
+ WorkerState *entry;
LogicalRepCommitData commit_data;
- MemoryContext oldcxt;
-
- Assert(!in_streamed_transaction);
-
- xid = logicalrep_read_stream_commit(s, &commit_data);
-
- elog(DEBUG1, "received commit for streamed transaction %u", xid);
-
- /* open the spool file for the committed transaction */
- changes_filename(path, MyLogicalRepWorker->subid, xid);
-
- elog(DEBUG1, "replaying changes from file '%s'", path);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
+ if (isLogicalApplyWorker)
{
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- }
-
- /* XXX Should this be allocated in another memory context? */
+ // logicalrep_read_stream_commit(s, &commit_data);
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- buffer = palloc(8192);
- initStringInfo(&s2);
-
- MemoryContextSwitchTo(oldcxt);
-
- ensure_transaction();
-
- /*
- * Make sure the handle apply_dispatch methods are aware we're in a remote
- * transaction.
- */
- in_remote_transaction = true;
- pgstat_report_activity(STATE_RUNNING, NULL);
-
- /*
- * Read the entries one by one and pass them through the same logic as in
- * apply_dispatch.
- */
- nchanges = 0;
- while (true)
+ CommitTransactionCommand();
+ }
+ else
{
- int nbytes;
- int len;
-
- /* read length of the on-disk record */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- nbytes = read(fd, &len, sizeof(len));
- pgstat_report_wait_end();
-
- /* have we reached end of the file? */
- if (nbytes == 0)
- break;
-
- /* do we have a correct length? */
- if (nbytes != sizeof(len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
-
- Assert(len > 0);
+ char action = 'F';
- /* make sure we have sufficiently large buffer */
- buffer = repalloc(buffer, len);
-
- /* and finally read the data into the buffer */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- if (read(fd, buffer, len) != len)
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
- pgstat_report_wait_end();
+ Assert(!in_streamed_transaction);
- /* copy the buffer to the stringinfo and call apply_dispatch */
- resetStringInfo(&s2);
- appendBinaryStringInfo(&s2, buffer, len);
+ xid = pq_getmsgint(s, 4);
+ logicalrep_read_stream_commit(s, &commit_data);
- /* Ensure we are reading the data into our memory context. */
- oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
- apply_dispatch(&s2);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(xid, false);
- MemoryContextReset(ApplyMessageContext);
+ /* Send commit message */
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- MemoryContextSwitchTo(oldcxt);
+ /* Notify worker, that we are done with this xact */
+ shm_mq_send(entry->mq_handle, 1, &action, false);
- nchanges++;
+ wait_for_worker_to_finish(entry);
- if (nchanges % 1000 == 0)
- elog(DEBUG1, "replayed %d changes from file '%s'",
- nchanges, path);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
/*
- * send feedback to upstream
- *
- * XXX Probably should send a valid LSN. But which one?
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
*/
- send_feedback(InvalidXLogRecPtr, false, false);
- }
-
- CloseTransientFile(fd);
-
- /*
- * Update origin state so we can restart streaming from correct
- * position in case of crash.
- */
- replorigin_session_origin_lsn = commit_data.end_lsn;
- replorigin_session_origin_timestamp = commit_data.committime;
-
- CommitTransactionCommand();
- pgstat_report_stat(false);
-
- store_flush_position(commit_data.end_lsn);
-
- elog(DEBUG1, "replayed %d (all) changes from file '%s'",
- nchanges, path);
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
- in_remote_transaction = false;
- pgstat_report_activity(STATE_IDLE, NULL);
+ pgstat_report_stat(false);
- /* unlink the files with serialized changes and subxact info */
- stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ store_flush_position(commit_data.end_lsn);
- pfree(buffer);
- pfree(s2.data);
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
}
/*
@@ -965,6 +971,8 @@ apply_handle_relation(StringInfo s)
if (handle_streamed_transaction('R', s))
return;
+ // iter_sleep(3600);
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -1407,6 +1415,38 @@ apply_dispatch(StringInfo s)
{
char action = pq_getmsgbyte(s);
+ if (isLogicalApplyWorker)
+ {
+ /*
+ * Inside logical apply worker we can figure out that new subtransaction
+ * was started if new change arrived with different xid. In that case we
+ * can define named savepoint, so that we were able to commit/rollback it
+ * separately later.
+ *
+ * Special case is if the first change comes from subtransuction, then
+ * we check that current_xid differs from stream_xid.
+ */
+ current_xid = pq_getmsgint(s, 4);
+
+ if (current_xid != stream_xid
+ && ((TransactionIdIsValid(prev_xid) && current_xid != prev_xid)
+ || !TransactionIdIsValid(prev_xid)))
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", current_xid);
+
+ elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname);
+
+ DefineSavepoint(spname);
+ CommitTransactionCommand();
+ // BeginInternalSubTransaction(NULL);
+ }
+
+ prev_xid = current_xid;
+ }
+ // else
+ // elog(LOG, "Logical worker: applying dispatch for action=%s", (char *) &action);
+
switch (action)
{
/* BEGIN */
@@ -1435,6 +1475,7 @@ apply_dispatch(StringInfo s)
break;
/* RELATION */
case 'R':
+ // elog(LOG, "%s worker: applying dispatch for action=R", isLogicalApplyWorker ? "Apply" : "Logical");
apply_handle_relation(s);
break;
/* TYPE */
@@ -1565,12 +1606,18 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
static void
worker_onexit(int code, Datum arg)
{
- int i;
+ HASH_SEQ_STATUS status;
+ WorkerState *entry;
- elog(LOG, "cleanup files for %d transactions", nxids);
-
- for (i = nxids-1; i >= 0; i--)
- stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+ if (ApplyWorkersHash != NULL)
+ {
+ hash_seq_init(&status, ApplyWorkersHash);
+ while ((entry = (WorkerState *) hash_seq_search(&status)) != NULL)
+ {
+ stop_worker(entry);
+ }
+ hash_seq_term(&status);
+ }
}
/*
@@ -1593,6 +1640,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
+ ApplyWorkersIdleList = palloc(sizeof(WorkerState *) * pool_size);
+
for (;;)
{
pgsocket fd = PGINVALID_SOCKET;
@@ -1904,8 +1953,9 @@ maybe_reread_subscription(void)
Subscription *newsub;
bool started_tx = false;
+ // TODO Probably we have to handle subscription reread in apply workers too.
/* When cache state is valid there is nothing to do here. */
- if (MySubscriptionValid)
+ if (MySubscriptionValid || isLogicalApplyWorker)
return;
/* This function might be called inside or outside of transaction. */
@@ -2039,608 +2089,50 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
-/*
- * subxact_info_write
- * Store information about subxacts for a toplevel transaction.
- *
- * For each subxact we store offset of it's first change in the main file.
- * The file is always over-written as a whole, and we also include CRC32C
- * checksum of the information.
- *
- * XXX We should only store subxacts that were not aborted yet.
- *
- * XXX Maybe we should only include the checksum when the cluster is
- * initialized with checksums?
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- */
+/* SIGHUP: set flag to reload configuration at next convenient time */
static void
-subxact_info_write(Oid subid, TransactionId xid)
+logicalrep_worker_sighup(SIGNAL_ARGS)
{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- Size len;
-
- Assert(TransactionIdIsValid(xid));
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create file \"%s\": %m",
- path)));
- return;
- }
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* compute the checksum */
- INIT_CRC32C(checksum);
- COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum, (char *) subxacts, len);
- FIN_CRC32C(checksum);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
-
- if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
+ int save_errno = errno;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ got_SIGHUP = true;
- if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ errno = save_errno;
+}
- if ((len > 0) && (write(fd, subxacts, len) != len))
- {
- int save_errno = errno;
+/* Logical Replication Apply worker entry point */
+void
+ApplyWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ MemoryContext oldctx;
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos;
+ char *myslotname;
+ WalRcvStreamOptions options;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
- pgstat_report_wait_end();
+ /* Setup signal handling */
+ pqsignal(SIGHUP, logicalrep_worker_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
/*
- * We don't need to fsync or anything, as we'll recreate the files after a
- * crash from scratch. So just close the file.
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
*/
- CloseTransientFile(fd);
- /*
- * But we free the memory allocated for subxact info. There might be one
- * exceptional transaction with many subxacts, and we don't want to keep
- * the memory allocated forewer.
- *
- */
- if (subxacts)
- pfree(subxacts);
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
- subxacts = NULL;
- subxact_last = InvalidTransactionId;
- nsubxacts = 0;
- nsubxacts_max = 0;
-}
-
-/*
- * subxact_info_read
- * Restore information about subxacts of a streamed transaction.
- *
- * Read information about subxacts into the global variables, and while
- * reading the information verify the checksum.
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_read(Oid subid, TransactionId xid)
-{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- uint32 checksum_new;
- Size len;
- MemoryContext oldctx;
-
- Assert(TransactionIdIsValid(xid));
- Assert(!subxacts);
- Assert(nsubxacts == 0);
- Assert(nsubxacts_max == 0);
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- /* read the checksum */
- if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- /* read number of subxact items */
- if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* we keep the maximum as a power of 2 */
- nsubxacts_max = 1 << my_log2(nsubxacts);
-
- /* subxacts are long-lived */
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- if ((len > 0) && ((read(fd, subxacts, len)) != len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- /* recompute the checksum */
- INIT_CRC32C(checksum_new);
- COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum_new, (char *) subxacts, len);
- FIN_CRC32C(checksum_new);
-
- if (checksum_new != checksum)
- ereport(ERROR,
- (errmsg("checksum failure when reading subxacts")));
-
- CloseTransientFile(fd);
-}
-
-/*
- * subxact_info_add
- * Add information about a subxact (offset in the main file).
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_add(TransactionId xid)
-{
- int64 i;
-
- /*
- * If the XID matches the toplevel transaction, we don't want to add it.
- */
- if (stream_xid == xid)
- return;
-
- /*
- * In most cases we're checking the same subxact as we've already seen in
- * the last call, so make ure just ignore it (this change comes later).
- */
- if (subxact_last == xid)
- return;
-
- /* OK, remember we're processing this XID. */
- subxact_last = xid;
-
- /*
- * Check if the transaction is already present in the array of subxact. We
- * intentionally scan the array from the tail, because we're likely adding
- * a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- */
- for (i = nsubxacts; i > 0; i--)
- {
- /* found, so we're done */
- if (subxacts[i - 1].xid == xid)
- return;
- }
-
- /* This is a new subxact, so we need to add it to the array. */
-
- if (nsubxacts == 0)
- {
- MemoryContext oldctx;
-
- nsubxacts_max = 128;
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
- }
- else if (nsubxacts == nsubxacts_max)
- {
- nsubxacts_max *= 2;
- subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
- }
-
- subxacts[nsubxacts].xid = xid;
- subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
-
- nsubxacts++;
-}
-
-/* format filename for file containing the info about subxacts */
-static void
-subxact_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
- tempdirpath, subid, xid);
-}
-
-/* format filename for file containing serialized changes */
-static void
-changes_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
- tempdirpath, subid, xid);
-}
-
-/*
- * stream_cleanup_files
- * Cleanup files for a subscription / toplevel transaction.
- *
- * Remove files with serialized changes and subxact info for a particular
- * toplevel transaction. Each subscription has a separate set of files.
- *
- * Note: The files may not exists, so handle ENOENT as non-error.
- *
- * TODO: Add missing_ok flag to specify in which cases it's OK not to
- * find the files, and when it's an error.
- */
-static void
-stream_cleanup_files(Oid subid, TransactionId xid)
-{
- int i;
- char path[MAXPGPATH];
- bool found = false;
-
- subxact_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- changes_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- /*
- * Cleanup the XID from the array - find the XID in the array and
- * remove it by shifting all the remaining elements. The array is
- * bound to be fairly small (maximum number of in-progress xacts,
- * so max_connections + max_prepared_transactions) so simply loop
- * through the array and find index of the XID. Then move the rest
- * of the array by one element to the left.
- *
- * Notice we also call this from stream_open_file for first segment
- * of each transaction, to deal with possible left-overs after a
- * crash, so it's entirely possible not to find the XID in the
- * array here. In that case we don't remove anything.
- *
- * XXX Perhaps it'd be better to handle this automatically after a
- * restart, instead of doing it over and over for each transaction.
- */
- for (i = 0; i < nxids; i++)
- {
- if (xids[i] == xid)
- {
- found = true;
- break;
- }
- }
-
- if (!found)
- return;
-
- /*
- * Move the last entry from the array to the place. We don't keep
- * the streamed transactions sorted or anything - we only expect
- * a few of them in progress (max_connections + max_prepared_xacts)
- * so linear search is just fine.
- */
- xids[i] = xids[nxids-1];
- nxids--;
-}
-
-/*
- * stream_open_file
- * Open file we'll use to serialize changes for a toplevel transaction.
- *
- * Open a file for streamed changes from a toplevel transaction identified
- * by stream_xid (global variable). If it's the first chunk of streamed
- * changes for this transaction, perform cleanup by removing existing
- * files after a possible previous crash.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_open_file(Oid subid, TransactionId xid, bool first_segment)
-{
- char path[MAXPGPATH];
- int flags;
-
- Assert(in_streamed_transaction);
- Assert(OidIsValid(subid));
- Assert(TransactionIdIsValid(xid));
- Assert(stream_fd == -1);
-
- /*
- * If this is the first segment for this transaction, try removing
- * existing files (if there are any, possibly after a crash).
- */
- if (first_segment)
- {
- MemoryContext oldcxt;
-
- /* XXX make sure there are no previous files for this transaction */
- stream_cleanup_files(subid, xid);
-
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- /*
- * We need to remember the XIDs we spilled to files, so that we can
- * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
- *
- * The number of XIDs we may need to track is fairly small, because
- * we can only stream toplevel xacts (so limited by max_connections
- * and max_prepared_transactions), and we only stream the large ones.
- * So we simply keep the XIDs in an unsorted array. If the number of
- * xacts gets large for some reason (e.g. very high max_connections),
- * a more elaborate approach might be better - e.g. sorted array, to
- * speed-up the lookups.
- */
- if (nxids == maxnxids) /* array of XIDs is full */
- {
- if (!xids)
- {
- maxnxids = 64;
- xids = palloc(maxnxids * sizeof(TransactionId));
- }
- else
- {
- maxnxids = 2 * maxnxids;
- xids = repalloc(xids, maxnxids * sizeof(TransactionId));
- }
- }
-
- xids[nxids++] = xid;
-
- MemoryContextSwitchTo(oldcxt);
- }
-
- changes_filename(path, subid, xid);
-
- elog(DEBUG1, "opening file '%s' for streamed changes", path);
-
- /*
- * If this is the first streamed segment, the file must not exist, so
- * make sure we're the ones creating it. Otherwise just open the file
- * for writing, in append mode.
- */
- if (first_segment)
- flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
- else
- flags = (O_WRONLY | O_APPEND | PG_BINARY);
-
- stream_fd = OpenTransientFile(path, flags);
-
- if (stream_fd < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
-}
-
-/*
- * stream_close_file
- * Close the currently open file with streamed changes.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_close_file(void)
-{
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- CloseTransientFile(stream_fd);
-
- stream_xid = InvalidTransactionId;
- stream_fd = -1;
-}
-
-/*
- * stream_write_change
- * Serialize a change to a file for the current toplevel transaction.
- *
- * The change is serialied in a simple format, with length (not including
- * the length), action code (identifying the message type) and message
- * contents (without the subxact TransactionId value).
- *
- * XXX The subxact file includes CRC32C of the contents. Maybe we should
- * include something like that here too, but doing so will not be as
- * straighforward, because we write the file in chunks.
- */
-static void
-stream_write_change(char action, StringInfo s)
-{
- int len;
-
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- /* total on-disk size, including the action type character */
- len = (s->len - s->cursor) + sizeof(char);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
-
- /* first write the size */
- if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* then the action */
- if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* and finally the remaining part of the buffer (after the XID) */
- len = (s->len - s->cursor);
-
- if (write(stream_fd, &s->data[s->cursor], len) != len)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- pgstat_report_wait_end();
-}
-
-/* SIGHUP: set flag to reload configuration at next convenient time */
-static void
-logicalrep_worker_sighup(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- got_SIGHUP = true;
-
- /* Waken anything waiting on the process latch */
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
-/* Logical Replication Apply worker entry point */
-void
-ApplyWorkerMain(Datum main_arg)
-{
- int worker_slot = DatumGetInt32(main_arg);
- MemoryContext oldctx;
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos;
- char *myslotname;
- WalRcvStreamOptions options;
-
- /* Attach to slot */
- logicalrep_worker_attach(worker_slot);
-
- /* Setup signal handling */
- pqsignal(SIGHUP, logicalrep_worker_sighup);
- pqsignal(SIGTERM, die);
- BackgroundWorkerUnblockSignals();
-
- /*
- * We don't currently need any ResourceOwner in a walreceiver process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
-
- /* Initialise stats to a sanish value */
- MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
- MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
-
- /* Load the libpq-specific functions */
- load_file("libpqwalreceiver", false);
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
/* Run as replica session replication role. */
SetConfigOption("session_replication_role", "replica",
@@ -2798,3 +2290,580 @@ IsLogicalWorker(void)
{
return MyLogicalRepWorker != NULL;
}
+
+/*
+ * Apply Background Worker main loop.
+ */
+void
+LogicalApplyBgwMain(Datum main_arg)
+{
+ volatile ParallelState *pst;
+
+ dsm_segment *seg;
+ shm_toc *toc;
+ PGPROC *registrant;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ shm_mq_result shmq_res;
+ // ConditionVariable cv;
+ LogicalRepWorker lrw;
+ MemoryContext oldcontext;
+
+ MemoryContextSwitchTo(TopMemoryContext);
+
+ /* Load the subscription into persistent memory context. */
+ ApplyContext = AllocSetContextCreate(TopMemoryContext,
+ "ApplyContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldcontext = MemoryContextSwitchTo(ApplyContext);
+
+ /*
+ * Init the ApplyMessageContext which we clean up after each replication
+ * protocol message.
+ */
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ isLogicalApplyWorker = true;
+
+ /*
+ * Establish signal handlers.
+ *
+ * We want CHECK_FOR_INTERRUPTS() to kill off this worker process just as
+ * it would a normal user backend. To make that happen, we establish a
+ * signal handler that is a stripped-down version of die().
+ */
+ pqsignal(SIGTERM, handle_sigterm);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Connect to the dynamic shared memory segment.
+ *
+ * The backend that registered this worker passed us the ID of a shared
+ * memory segment to which we must attach for further instructions. In
+ * order to attach to dynamic shared memory, we need a resource owner.
+ * Once we've mapped the segment in our address space, attach to the table
+ * of contents so we can locate the various data structures we'll need to
+ * find within the segment.
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker");
+ seg = dsm_attach(DatumGetInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("unable to map dynamic shared memory segment")));
+ toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg));
+ if (toc == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("bad magic number in dynamic shared memory segment")));
+
+ /*
+ * Acquire a worker number.
+ *
+ * By convention, the process registering this background worker should
+ * have stored the control structure at key 0. We look up that key to
+ * find it. Our worker number gives our identity: there may be just one
+ * worker involved in this parallel operation, or there may be many.
+ */
+ pst = shm_toc_lookup(toc, 0, false);
+ MyParallelState = pst;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->attached = true;
+ SpinLockRelease(&pst->mutex);
+
+ /*
+ * Attach to the message queue.
+ */
+ mq = shm_toc_lookup(toc, 1, false);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ /* Restore database connection. */
+ BackgroundWorkerInitializeConnectionByOid(pst->database_id,
+ pst->authenticated_user_id, 0);
+
+ /*
+ * Set the client encoding to the database encoding, since that is what
+ * the leader will expect.
+ */
+ SetClientEncoding(GetDatabaseEncoding());
+
+ lrw.subid = pst->subid;
+ MyLogicalRepWorker = &lrw;
+
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+ // PushActiveSnapshot(GetTransactionSnapshot());
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ /*
+ * Indicate that we're fully initialized and ready to begin the main part
+ * of the parallel operation.
+ *
+ * Once we signal that we're ready, the user backend is entitled to assume
+ * that our on_dsm_detach callbacks will fire before we disconnect from
+ * the shared memory segment and exit. Generally, that means we must have
+ * attached to all relevant dynamic shared memory data structures by now.
+ */
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ // cv = pst->cv;
+ // if (pst->workers_ready == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+ elog(LOG, "[Apply BGW #%u] started", pst->n);
+
+ registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ SetLatch(®istrant->procLatch);
+
+ for (;;)
+ {
+ void *data;
+ Size len;
+ StringInfoData s;
+ MemoryContext oldctx;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Ensure we are reading the data into our memory context. */
+ oldctx = MemoryContextSwitchTo(ApplyMessageContext);
+
+ shmq_res = shm_mq_receive(mqh, &len, &data, false);
+
+ if (shmq_res != SHM_MQ_SUCCESS)
+ break;
+
+ if (len == 0)
+ {
+ elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n);
+ break;
+ }
+ else
+ {
+ s.cursor = 0;
+ s.maxlen = -1;
+ s.data = (char *) data;
+ s.len = len;
+
+ /*
+ * We use first byte of message for additional communication between
+ * main Logical replication worker and Apply BGWorkers, so if it
+ * differs from 'w', then process it first.
+ */
+ switch (pq_getmsgbyte(&s))
+ {
+ /* Stream stop */
+ case 'E':
+ {
+ in_remote_transaction = false;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ SpinLockRelease(&pst->mutex);
+ SetLatch(®istrant->procLatch);
+
+ elog(LOG, "[Apply BGW #%u] ended processing streaming chunk, waiting on shm_mq_receive", pst->n);
+
+ continue;
+ }
+ /* Reassign to the new transaction */
+ case 'R':
+ {
+ elog(LOG, "[Apply BGW #%u] switching from processing xid %u to xid %u",
+ pst->n, stream_xid, pst->stream_xid);
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ continue;
+ }
+ /* Finished processing xact */
+ case 'F':
+ {
+ elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid);
+
+ MemoryContextSwitchTo(ApplyContext);
+
+ CommitTransactionCommand();
+ EndTransactionBlock();
+ CommitTransactionCommand();
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ SpinLockRelease(&pst->mutex);
+
+ continue;
+ }
+ default:
+ break;
+ }
+
+ pq_getmsgint64(&s); // Read LSN info
+ pq_getmsgint64(&s); // TODO Do we need to process it here again somehow?
+ pq_getmsgint64(&s);
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ elog(DEBUG5, "[Apply BGW #%u] applying dispatch for action=%s",
+ pst->n, (char *) &s.data[s.cursor]);
+ apply_dispatch(&s);
+ }
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextReset(ApplyMessageContext);
+ }
+
+ CommitTransactionCommand();
+ EndTransactionBlock();
+ CommitTransactionCommand();
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(ApplyContext);
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ // if (pst->workers_finished == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+
+ elog(LOG, "[Apply BGW #%u] exiting", pst->n);
+
+ /* Signal main process that we are done. */
+ // ConditionVariableBroadcast(&cv);
+ SetLatch(®istrant->procLatch);
+
+ /*
+ * We're done. Explicitly detach the shared memory segment so that we
+ * don't get a resource leak warning at commit time. This will fire any
+ * on_dsm_detach callbacks we've registered, as well. Once that's done,
+ * we can go ahead and exit.
+ */
+ dsm_detach(seg);
+ proc_exit(0);
+}
+
+/*
+ * When we receive a SIGTERM, we set InterruptPending and ProcDiePending just
+ * like a normal backend. The next CHECK_FOR_INTERRUPTS() will do the right
+ * thing.
+ */
+static void
+handle_sigterm(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ SetLatch(MyLatch);
+
+ if (!proc_exit_inprogress)
+ {
+ InterruptPending = true;
+ ProcDiePending = true;
+ }
+
+ errno = save_errno;
+}
+
+/*
+ * Set up a dynamic shared memory segment.
+ *
+ * We set up a control region that contains a ParallelState,
+ * plus one region per message queue. There are as many message queues as
+ * the number of workers.
+ */
+static void
+setup_dsm(WorkerState *wstate)
+{
+ shm_toc_estimator e;
+ int toc_key = 0;
+ Size segsize;
+ dsm_segment *seg;
+ shm_toc *toc;
+ ParallelState *pst;
+ shm_mq *mq;
+ int64 queue_size = 160000000; /* 16 MB for now */
+
+ /* Ensure a valid queue size. */
+ if (queue_size < 0 || ((uint64) queue_size) < shm_mq_minimum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size must be at least %zu bytes",
+ shm_mq_minimum_size)));
+ if (queue_size != ((Size) queue_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size overflows size_t")));
+
+ /*
+ * Estimate how much shared memory we need.
+ *
+ * Because the TOC machinery may choose to insert padding of oddly-sized
+ * requests, we must estimate each chunk separately.
+ *
+ * We need one key to register the location of the header, and we need
+ * nworkers keys to track the locations of the message queues.
+ */
+ shm_toc_initialize_estimator(&e);
+ shm_toc_estimate_chunk(&e, sizeof(ParallelState));
+ shm_toc_estimate_chunk(&e, (Size) queue_size);
+
+ shm_toc_estimate_keys(&e, 1 + 1);
+ segsize = shm_toc_estimate(&e);
+
+ /* Create the shared memory segment and establish a table of contents. */
+ seg = dsm_create(shm_toc_estimate(&e), 0);
+ toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg),
+ segsize);
+
+ /* Set up the header region. */
+ pst = shm_toc_allocate(toc, sizeof(ParallelState));
+ SpinLockInit(&pst->mutex);
+ pst->attached = false;
+ pst->ready = false;
+ pst->finished = false;
+ pst->database_id = MyDatabaseId;
+ pst->subid = MyLogicalRepWorker->subid;
+ pst->stream_xid = stream_xid;
+ pst->authenticated_user_id = GetAuthenticatedUserId();
+ pst->n = nworkers + 1;
+ // ConditionVariableInit(&pst->cv);
+
+ shm_toc_insert(toc, toc_key++, pst);
+
+ /* Set up one message queue per worker, plus one. */
+ mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size),
+ (Size) queue_size);
+ shm_toc_insert(toc, toc_key++, mq);
+ shm_mq_set_sender(mq, MyProc);
+
+ /* Attach the queues. */
+ wstate->mq_handle = shm_mq_attach(mq, seg, wstate->handle);
+
+ /* Return results to caller. */
+ wstate->dsm_seg = seg;
+ wstate->pstate = pst;
+}
+
+/*
+ * Register background workers.
+ */
+static void
+setup_background_worker(WorkerState *wstate)
+{
+ MemoryContext oldcontext;
+ BackgroundWorker worker;
+
+ elog(LOG, "setting up apply worker #%u", nworkers + 1);
+
+ /*
+ * TOCHECK: We need the worker_state object and the background worker handles to
+ * which it points to be allocated in TopMemoryContext rather than
+ * ApplyMessageContext; otherwise, they'll be destroyed before the on_dsm_detach
+ * hooks run.
+ */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+
+ setup_dsm(wstate);
+
+ /*
+ * Arrange to kill all the workers if we abort before all workers are
+ * finished hooking themselves up to the dynamic shared memory segment.
+ *
+ * If we die after all the workers have finished hooking themselves up to
+ * the dynamic shared memory segment, we'll mark the two queues to which
+ * we're directly connected as detached, and the worker(s) connected to
+ * those queues will exit, marking any other queues to which they are
+ * connected as detached. This will cause any as-yet-unaware workers
+ * connected to those queues to exit in their turn, and so on, until
+ * everybody exits.
+ *
+ * But suppose the workers which are supposed to connect to the queues to
+ * which we're directly attached exit due to some error before they
+ * actually attach the queues. The remaining workers will have no way of
+ * knowing this. From their perspective, they're still waiting for those
+ * workers to start, when in fact they've already died.
+ */
+ on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ /* Configure a worker. */
+ MemSet(&worker, 0, sizeof(BackgroundWorker));
+
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_ConsistentState;
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_notify_pid = MyProcPid;
+ sprintf(worker.bgw_library_name, "postgres");
+ sprintf(worker.bgw_function_name, "LogicalApplyBgwMain");
+
+ worker.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(wstate->dsm_seg));
+
+ /* Register the workers. */
+ snprintf(worker.bgw_name, BGW_MAXLEN,
+ "logical replication apply worker #%u for subscription %u",
+ nworkers + 1, MySubscription->oid);
+ if (!RegisterDynamicBackgroundWorker(&worker, &wstate->handle))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not register background process"),
+ errhint("You may need to increase max_worker_processes.")));
+
+ /* All done. */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Wait for worker to become ready. */
+ wait_for_worker(wstate);
+
+ /*
+ * Once we reach this point, all workers are ready. We no longer need to
+ * kill them if we die; they'll die on their own as the message queues
+ * shut down.
+ */
+ cancel_on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ nworkers += 1;
+}
+
+static void
+cleanup_background_worker(dsm_segment *seg, Datum arg)
+{
+ WorkerState *wstate = (WorkerState *) DatumGetPointer(arg);
+
+ TerminateBackgroundWorker(wstate->handle);
+}
+
+static void
+wait_for_worker(WorkerState *wstate)
+{
+ bool result = false;
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool ready;
+
+ /* If the worker is ready, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ ready = wstate->pstate->ready;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (ready)
+ {
+ result = true;
+ break;
+ }
+
+ /* If any workers (or the postmaster) have died, we have failed. */
+ if (!check_worker_status(wstate))
+ {
+ result = false;
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ // ConditionVariableCancelSleep();
+
+ if (!result)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("one or more background workers failed to start")));
+}
+
+static bool
+check_worker_status(WorkerState *wstate)
+{
+ BgwHandleStatus status;
+ pid_t pid;
+
+ status = GetBackgroundWorkerPid(wstate->handle, &pid);
+ if (status == BGWH_STOPPED || status == BGWH_POSTMASTER_DIED)
+ return false;
+
+ /* Otherwise, things still look OK. */
+ return true;
+}
+
+static void
+wait_for_worker_to_finish(WorkerState *wstate)
+{
+ elog(LOG, "waiting for apply worker #%u to finish processing xid %u",
+ wstate->pstate->n, wstate->xid);
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool finished;
+
+ /* If the worker is finished, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ finished = wstate->pstate->finished;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (finished)
+ {
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3a89e23488..7c72db9e83 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -819,6 +819,7 @@ typedef enum
WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATING,
WAIT_EVENT_HASH_GROW_BUCKETS_ELECTING,
WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY,
WAIT_EVENT_LOGICAL_SYNC_DATA,
WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
WAIT_EVENT_MQ_INTERNAL,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 802275311d..afb15c2736 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -122,12 +122,10 @@ extern TransactionId logicalrep_read_stream_stop(StringInfo in);
extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn);
-extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+extern void logicalrep_read_stream_commit(StringInfo out,
LogicalRepCommitData *commit_data);
extern void logicalrep_write_stream_abort(StringInfo out,
TransactionId xid, TransactionId subxid);
-extern void logicalrep_read_stream_abort(StringInfo in,
- TransactionId *xid, TransactionId *subxid);
#endif /* LOGICALREP_PROTO_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index e9524aefd9..30ad40247d 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -13,6 +13,7 @@
#define LOGICALWORKER_H
extern void ApplyWorkerMain(Datum main_arg);
+extern void LogicalApplyBgwMain(Datum main_arg);
extern bool IsLogicalWorker(void);
--
2.17.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-16 19:29 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Konstantin Knizhnik @ 2019-09-16 19:29 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On 16.09.2019 19:54, Alexey Kondratov wrote:
> On 30.08.2019 18:59, Konstantin Knizhnik wrote:
>>
>> I think that instead of defining savepoints it is simpler and more
>> efficient to use
>>
>> BeginInternalSubTransaction +
>> ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction
>>
>> as it is done in PL/pgSQL (pl_exec.c).
>> Not sure if it can pr
>>
>
> Both BeginInternalSubTransaction and DefineSavepoint use
> PushTransaction() internally for a normal subtransaction start. So
> they seems to be identical from the performance perspective, which is
> also stated in the comment section:
Yes, definitely them are using the same mechanism and most likely
provides similar performance.
But BeginInternalSubTransaction does not require to generate some
savepoint name which seems to be redundant in this case.
>
> Anyway, I've performed a profiling of my apply worker (flamegraph is
> attached) and it spends the vast amount of time (>90%) applying
> changes. So the problem is not in the savepoints their-self, but in
> the fact that we first apply all changes and then abort all the work.
> Not sure, that it is possible to do something in this case.
>
Looks like the only way to increase apply speed is to do it in parallel:
make it possible to concurrently execute non-conflicting transactions.
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-16 20:57 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-16 20:57 UTC (permalink / raw)
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; +Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Erik Rijkers <er@xs4all.nl>; Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers
On Mon, Sep 16, 2019 at 10:29:18PM +0300, Konstantin Knizhnik wrote:
>
>
>On 16.09.2019 19:54, Alexey Kondratov wrote:
>>On 30.08.2019 18:59, Konstantin Knizhnik wrote:
>>>
>>>I think that instead of defining savepoints it is simpler and more
>>>efficient to use
>>>
>>>BeginInternalSubTransaction +
>>>ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction
>>>
>>>as it is done in PL/pgSQL (pl_exec.c).
>>>Not sure if it can pr
>>>
>>
>>Both BeginInternalSubTransaction and DefineSavepoint use
>>PushTransaction() internally for a normal subtransaction start. So
>>they seems to be identical from the performance perspective, which
>>is also stated in the comment section:
>
>Yes, definitely them are using the same mechanism and most likely
>provides similar performance.
>But BeginInternalSubTransaction does not require to generate some
>savepoint name which seems to be redundant in this case.
>
>
>>
>>Anyway, I've performed a profiling of my apply worker (flamegraph is
>>attached) and it spends the vast amount of time (>90%) applying
>>changes. So the problem is not in the savepoints their-self, but in
>>the fact that we first apply all changes and then abort all the
>>work. Not sure, that it is possible to do something in this case.
>>
>
>Looks like the only way to increase apply speed is to do it in
>parallel: make it possible to concurrently execute non-conflicting
>transactions.
>
True, although it seems like a massive can of worms to me. I'm not aware
a way to identify non-conflicting transactions in advance, so it would
have to be implemented as optimistic apply, with a detection and
recovery from conflicts.
I'm not against doing that, and I'm willing to spend some time on revies
etc. but it seems like a completely separate effort.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-25 13:25 Amit Kapila <amit.kapila16@gmail.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-25 13:25 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On Tue, Sep 3, 2019 at 4:30 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
> In the interest of moving things forward, how far are we from making
> 0001 committable? If I understand correctly, the rest of this patchset
> depends on https://commitfest.postgresql.org/24/944/ which seems to be
> moving at a glacial pace (or, actually, slower, because glaciers do
> move, which cannot be said of that other patch.)
>
I am not sure if it is completely correct that the other part of the
patch is dependent on that CF entry. I have studied both the threads
(not every detail) and it seems to me it is dependent on one of the
patches from that series which handles concurrent aborts. It is patch
0003-Gracefully-handle-concurrent-aborts-of-uncommitted-t.Jan4.patch
from what the Nikhil has posted on that thread [1]. Am, I wrong?
So IIUC, the problem of concurrent aborts is that if we allow catalog
scans for in-progress transactions, then we might get wrong answers in
cases where somebody has performed Alter-Abort-Alter which is clearly
explained with an example in email [2]. To solve that problem Nikhil
seems to have written a patch [1] which detects these concurrent
aborts during a system table scan and then aborts the decoding of such
a transaction.
Now, the problem is that patch has written considering 2PC
transactions and might not deal with all cases for in-progress
transactions especially when sub-transactions are involved as alluded
by Arseny Sher [3]. So, the problem seems to be for cases when some
sub-transaction aborts, but the main transaction still continued and
we try to decode it. Nikhil's patch won't be able to deal with it
because I think it just checks top-level xid whereas for this we need
to check all-subxids which I think is possible now as Tomas seems to
have written WAL for each xid-assignment. It might or might not be
the best solution to check the status of all-subxids, but I think
first we need to agree that the problem is just for concurrent aborts
and that we can solve it by using some part of the technology being
developed as part of patch "Logical decoding of two-phase
transactions" (https://commitfest.postgresql.org/24/944/) rather than
the entire patchset.
I hope I am not saying something very obvious here and it helps in
moving this patch forward.
Thoughts?
[1] - https://www.postgresql.org/message-id/CAMGcDxcBmN6jNeQkgWddfhX8HbSjQpW%3DUo70iBY3P_EPdp%2BLTQ%40mail...
[2] - https://www.postgresql.org/message-id/EEBD82AA-61EE-46F4-845E-05B94168E8F2%40postgrespro.ru
[3] - https://www.postgresql.org/message-id/87a7py4iwl.fsf%40ars-thinkpad
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 13:28 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-26 13:28 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Tue, Sep 3, 2019 at 4:16 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Mon, Sep 02, 2019 at 06:06:50PM -0400, Alvaro Herrera wrote:
> >In the interest of moving things forward, how far are we from making
> >0001 committable? If I understand correctly, the rest of this patchset
> >depends on https://commitfest.postgresql.org/24/944/ which seems to be
> >moving at a glacial pace (or, actually, slower, because glaciers do
> >move, which cannot be said of that other patch.)
> >
>
> I think 0001 is mostly there. I think there's one bug in this patch
> version, but I need to check and I'll post an updated version shortly if
> needed.
>
Did you get a chance to work on 0001? I have a few comments on that patch:
1.
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is either
+ * serialized or streamed.
Do we need to mention 'streamed' as part of this patch? It seems to
me that this is an independent patch which can be committed without
patches that stream the changes. So, we can remove it from here and
other places where it is used.
2.
+ * deserializing and applying very few changes). We probably to give more
+ * memory to the oldest subtransactions.
/We probably to/
It seems some word is missing after probably.
3.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
What is the guarantee that after evicting largest transaction, we
won't immediately hit the memory limit? Say, all of the transactions
are of almost similar size which I don't think is that uncommon a
case. Instead, the strategy mentioned in point (c) or something like
that seems more promising. In that strategy, there is some risk that
it might lead to many smaller disk writes which we might want to
control via some threshold (like we should not flush more than N
xacts). In this, we also need to ensure that the total memory freed
must be greater than the current change.
I think we have some discussion around this point but didn't reach any
conclusion which means some more brainstorming is required.
4.
+int logical_work_mem; /* 4MB */
What this 4MB in comments indicate?
5.
+/*
+ * Check whether the logical_work_mem limit was reached, and if yes pick
+ * the transaction tx should spill its data to disk.
The second part of the sentence "pick the transaction tx should spill"
seems to be incomplete.
Apart from this, I see that Peter E. has raised some other points on
this patch which are not yet addressed as those also need some
discussion, so I will respond to those separately with my opinion.
These comments are based on the last patch posted by you on this
thread [1]. You might have fixed some of these already, so ignore if
that is the case.
[1] - https://www.postgresql.org/message-id/76fc440e-91c3-afe2-b78a-987205b3c758%402ndquadrant.com
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 18:07 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 4 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-26 18:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
Hi,
Attached is an updated patch series, rebased on current master. It does
fix one memory accounting bug in ReorderBufferToastReplace (the code was
not properly updating the amount of memory).
I've also included the patch series with decoding of 2PC transactions,
which this depends on. This way we have a chance of making the cfbot
happy. So parts 0001-0004 and 0009-0014 are "this" patch series, while
0005-0008 are the extra pieces from the other patch.
I've done it like this because the initial parts are independent, and so
might be committed irrespectedly of the other patch series. In practice
that's only reasonable for 0001, which adds the memory limit - the rest
is infrastucture for the streaming of in-progress transactions.
On Wed, Sep 25, 2019 at 06:55:01PM +0530, Amit Kapila wrote:
>On Tue, Sep 3, 2019 at 4:30 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>>
>> In the interest of moving things forward, how far are we from making
>> 0001 committable? If I understand correctly, the rest of this patchset
>> depends on https://commitfest.postgresql.org/24/944/ which seems to be
>> moving at a glacial pace (or, actually, slower, because glaciers do
>> move, which cannot be said of that other patch.)
>>
>
>I am not sure if it is completely correct that the other part of the
>patch is dependent on that CF entry. I have studied both the threads
>(not every detail) and it seems to me it is dependent on one of the
>patches from that series which handles concurrent aborts. It is patch
>0003-Gracefully-handle-concurrent-aborts-of-uncommitted-t.Jan4.patch
>from what the Nikhil has posted on that thread [1]. Am, I wrong?
>
You're right - the part handling aborts is the only part required. There
are dependencies on some other changes from the 2PC patch, but those are
mostly refactorings that can be undone (e.g. switch from independent
flags to a single bitmap in reorderbuffer).
>So IIUC, the problem of concurrent aborts is that if we allow catalog
>scans for in-progress transactions, then we might get wrong answers in
>cases where somebody has performed Alter-Abort-Alter which is clearly
>explained with an example in email [2]. To solve that problem Nikhil
>seems to have written a patch [1] which detects these concurrent
>aborts during a system table scan and then aborts the decoding of such
>a transaction.
>
>Now, the problem is that patch has written considering 2PC
>transactions and might not deal with all cases for in-progress
>transactions especially when sub-transactions are involved as alluded
>by Arseny Sher [3]. So, the problem seems to be for cases when some
>sub-transaction aborts, but the main transaction still continued and
>we try to decode it. Nikhil's patch won't be able to deal with it
>because I think it just checks top-level xid whereas for this we need
>to check all-subxids which I think is possible now as Tomas seems to
>have written WAL for each xid-assignment. It might or might not be
>the best solution to check the status of all-subxids, but I think
>first we need to agree that the problem is just for concurrent aborts
>and that we can solve it by using some part of the technology being
>developed as part of patch "Logical decoding of two-phase
>transactions" (https://commitfest.postgresql.org/24/944/) rather than
>the entire patchset.
>
>I hope I am not saying something very obvious here and it helps in
>moving this patch forward.
>
No, that's a good question, and I'm not sure what the answer is at the
moment. My understanding was that the infrastructure in the 2PC patch is
enough even for subtransactions, but I might be wrong. I need to think
about that for a while.
Maybe we should focus on the 0001 part for now - it can be committed
indepently and does provide useful feature.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_work_mem-to-limit-ReorderBuffer-20190926.patch.gz (application/gzip, 10.4K) download
0002-Immediately-WAL-log-assignments-20190926.patch.gz (application/gzip, 6.6K) download
0003-Issue-individual-invalidations-with-wal_lev-20190926.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-me-20190926.patch.gz (application/gzip, 6.1K) download
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-st-20190926.patch.gz (application/gzip, 2.5K) download
0006-Support-decoding-of-two-phase-transactions--20190926.patch.gz (application/gzip, 9.0K) download
0007-Gracefully-handle-concurrent-aborts-of-unco-20190926.patch.gz (application/gzip, 4.4K) download
0008-Teach-test_decoding-plugin-to-work-with-2PC-20190926.patch.gz (application/gzip, 5.9K) download
0009-Implement-streaming-mode-in-ReorderBuffer-20190926.patch.gz (application/gzip, 12.7K) download
0010-Add-support-for-streaming-to-built-in-repli-20190926.patch.gz (application/gzip, 19.8K) download
0011-Track-statistics-for-streaming-spilling-20190926.patch.gz (application/gzip, 4.2K) download
0012-Enable-streaming-for-all-subscription-TAP-t-20190926.patch.gz (application/gzip, 2.1K) download
0013-BUGFIX-set-final_lsn-for-subxacts-before-cl-20190926.patch.gz (application/gzip, 631B) download
0014-Add-TAP-test-for-streaming-vs.-DDL-20190926.patch.gz (application/gzip, 1.6K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 18:36 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-26 18:36 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 06:58:17PM +0530, Amit Kapila wrote:
>On Tue, Sep 3, 2019 at 4:16 PM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Mon, Sep 02, 2019 at 06:06:50PM -0400, Alvaro Herrera wrote:
>> >In the interest of moving things forward, how far are we from making
>> >0001 committable? If I understand correctly, the rest of this patchset
>> >depends on https://commitfest.postgresql.org/24/944/ which seems to be
>> >moving at a glacial pace (or, actually, slower, because glaciers do
>> >move, which cannot be said of that other patch.)
>> >
>>
>> I think 0001 is mostly there. I think there's one bug in this patch
>> version, but I need to check and I'll post an updated version shortly if
>> needed.
>>
>
>Did you get a chance to work on 0001? I have a few comments on that patch:
>1.
>+ * To limit the amount of memory used by decoded changes, we track memory
>+ * used at the reorder buffer level (i.e. total amount of memory), and for
>+ * each toplevel transaction. When the total amount of used memory exceeds
>+ * the limit, the toplevel transaction consuming the most memory is either
>+ * serialized or streamed.
>
>Do we need to mention 'streamed' as part of this patch? It seems to
>me that this is an independent patch which can be committed without
>patches that stream the changes. So, we can remove it from here and
>other places where it is used.
>
You're right - this patch should not mention streaming because the parts
adding that capability are later in the series. So it can trigger just
the serialization to disk.
>2.
>+ * deserializing and applying very few changes). We probably to give more
>+ * memory to the oldest subtransactions.
>
>/We probably to/
>It seems some word is missing after probably.
>
Yes.
>3.
>+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
>+ *
>+ * XXX With many subtransactions this might be quite slow, because we'll have
>+ * to walk through all of them. There are some options how we could improve
>+ * that: (a) maintain some secondary structure with transactions sorted by
>+ * amount of changes, (b) not looking for the entirely largest transaction,
>+ * but e.g. for transaction using at least some fraction of the memory limit,
>+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
>+ * of the memory limit (e.g. 50%).
>+ */
>+static ReorderBufferTXN *
>+ReorderBufferLargestTXN(ReorderBuffer *rb)
>
>What is the guarantee that after evicting largest transaction, we
>won't immediately hit the memory limit? Say, all of the transactions
>are of almost similar size which I don't think is that uncommon a
>case.
Not sure I understand - what do you mean 'immediately hit'?
We do check the limit after queueing a change, and we know that this
change is what got us over the limit. We pick the largest transaction
(which has to be larger than the change we just entered) and evict it,
getting below the memory limit again.
The next change can get us over the memory limit again, of course, but
there's not much we could do about that.
> Instead, the strategy mentioned in point (c) or something like
>that seems more promising. In that strategy, there is some risk that
>it might lead to many smaller disk writes which we might want to
>control via some threshold (like we should not flush more than N
>xacts). In this, we also need to ensure that the total memory freed
>must be greater than the current change.
>
>I think we have some discussion around this point but didn't reach any
>conclusion which means some more brainstorming is required.
>
I agree it's worth investigating, but I'm not sure it's necessary before
committing v1 of the feature. I don't think there's a clear winner
strategy, and the current approach works fairly well I think.
The comment is concerned with the cost of ReorderBufferLargestTXN with
many transactions, but we can only have certain number of top-level
transactions (max_connections + certain number of not-yet-assigned
subtransactions). And 0002 patch essentially gets rid of the subxacts
entirely, further reducing the maximum number of xacts to walk.
>4.
>+int logical_work_mem; /* 4MB */
>
>What this 4MB in comments indicate?
>
Sorry, that's a mistake.
>5.
>+/*
>+ * Check whether the logical_work_mem limit was reached, and if yes pick
>+ * the transaction tx should spill its data to disk.
>
>The second part of the sentence "pick the transaction tx should spill"
>seems to be incomplete.
>
Yeah, that's a poor wording. Will fix.
>Apart from this, I see that Peter E. has raised some other points on
>this patch which are not yet addressed as those also need some
>discussion, so I will respond to those separately with my opinion.
>
OK, thanks.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 19:33 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
3 siblings, 2 replies; 536+ messages in thread
From: Alvaro Herrera @ 2019-09-26 19:33 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers
On 2019-Sep-26, Tomas Vondra wrote:
> Hi,
>
> Attached is an updated patch series, rebased on current master. It does
> fix one memory accounting bug in ReorderBufferToastReplace (the code was
> not properly updating the amount of memory).
Cool.
Can we aim to get 0001 pushed during this commitfest, or is that a lost
cause?
The large new comment in reorderbuffer.c says that a transaction might
get spilled *or streamed*, but surely that second thing is not correct,
since before the subsequent patches it's not possible to stream
transactions that have not yet finished?
How certain are you about the approach to measure memory used by a
reorderbuffer transaction ... does it not cause a measurable performance
drop? I wonder if it would make more sense to use a separate contexts
per transaction and use context-level accounting (per the patch Jeff
Davis posted elsewhere for hash joins ... though I see now that that
only works fot aset.c, not other memcxt implementations), or something
like that.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 19:36 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2019-09-26 19:36 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers
On 2019-Sep-26, Alvaro Herrera wrote:
> How certain are you about the approach to measure memory used by a
> reorderbuffer transaction ... does it not cause a measurable performance
> drop? I wonder if it would make more sense to use a separate contexts
> per transaction and use context-level accounting (per the patch Jeff
> Davis posted elsewhere for hash joins ... though I see now that that
> only works fot aset.c, not other memcxt implementations), or something
> like that.
Oh, I just noticed that that patch was posted separately in its own
thread, and that that improved version does include support for other
memory context implementations. Excellent.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 21:24 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-26 21:24 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 04:36:20PM -0300, Alvaro Herrera wrote:
>On 2019-Sep-26, Alvaro Herrera wrote:
>
>> How certain are you about the approach to measure memory used by a
>> reorderbuffer transaction ... does it not cause a measurable performance
>> drop? I wonder if it would make more sense to use a separate contexts
>> per transaction and use context-level accounting (per the patch Jeff
>> Davis posted elsewhere for hash joins ... though I see now that that
>> only works fot aset.c, not other memcxt implementations), or something
>> like that.
>
>Oh, I just noticed that that patch was posted separately in its own
>thread, and that that improved version does include support for other
>memory context implementations. Excellent.
>
Unfortunately, that won't fly, for two simple reasons:
1) The memory accounting patch is known to perform poorly with many
child contexts - this was why array_agg/string_agg were problematic,
before we rewrote them not to create memory context for each group.
It could be done differently (eager accounting) but then the overhead
for regular/common cases (with just a couple of contexts) is higher. So
that seems like a much inferior option.
2) We can't actually have a single context per transaction. Some parts
(REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID) of a transaction are not
evicted, so we'd have to keep them in a separate context.
It'd also mean higher allocation overhead, because now we can reuse
chunks cross-transaction. So one transaction commits or gets serialized,
and we reuse the chunks for something else. With per-transaction
contexts we'd lose some of this benefit - we could only reuse chunks
within a transaction (i.e. large transactions that get spilled to disk)
but not across commits.
I don't have any numbers, of course, but I wouldn't be surprised if it
was significant e.g. for small transactions that don't get spilled. And
creating/destroying the contexts is not free either, I think.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-26 21:46 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-26 21:46 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 04:33:59PM -0300, Alvaro Herrera wrote:
>On 2019-Sep-26, Tomas Vondra wrote:
>
>> Hi,
>>
>> Attached is an updated patch series, rebased on current master. It does
>> fix one memory accounting bug in ReorderBufferToastReplace (the code was
>> not properly updating the amount of memory).
>
>Cool.
>
>Can we aim to get 0001 pushed during this commitfest, or is that a lost
>cause?
>
It's tempting. The patch has been in the queue for quite a bit of time,
and I think it's solid (at least 0001). I'll address the comments from
Peter's review about separating the GUC etc. and polish it a bit more.
If I manage to do that by Monday, I'll consider pushing it.
If anyone feels I shouldn't do that, let me know.
The one open question pointed out by Amit is how the patch picks the
trasction for eviction. My feeling is that's fine and if needed can be
improved later if necessary, but I'll try to construct a worst case
(max_connections xacts, each with 64 subxact) to verify.
>The large new comment in reorderbuffer.c says that a transaction might
>get spilled *or streamed*, but surely that second thing is not correct,
>since before the subsequent patches it's not possible to stream
>transactions that have not yet finished?
>
True. That's a residue of reordering the patch series repeatedly, I
think. I'll fix that while polishing the patch.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-27 05:48 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-09-27 05:48 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Fri, Sep 27, 2019 at 12:06 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Thu, Sep 26, 2019 at 06:58:17PM +0530, Amit Kapila wrote:
>
> >3.
> >+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
> >+ *
> >+ * XXX With many subtransactions this might be quite slow, because we'll have
> >+ * to walk through all of them. There are some options how we could improve
> >+ * that: (a) maintain some secondary structure with transactions sorted by
> >+ * amount of changes, (b) not looking for the entirely largest transaction,
> >+ * but e.g. for transaction using at least some fraction of the memory limit,
> >+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
> >+ * of the memory limit (e.g. 50%).
> >+ */
> >+static ReorderBufferTXN *
> >+ReorderBufferLargestTXN(ReorderBuffer *rb)
> >
> >What is the guarantee that after evicting largest transaction, we
> >won't immediately hit the memory limit? Say, all of the transactions
> >are of almost similar size which I don't think is that uncommon a
> >case.
>
> Not sure I understand - what do you mean 'immediately hit'?
>
> We do check the limit after queueing a change, and we know that this
> change is what got us over the limit. We pick the largest transaction
> (which has to be larger than the change we just entered) and evict it,
> getting below the memory limit again.
>
> The next change can get us over the memory limit again, of course,
>
Yeah, this is what I want to say when I wrote that it can immediately hit again.
> but
> there's not much we could do about that.
>
> > Instead, the strategy mentioned in point (c) or something like
> >that seems more promising. In that strategy, there is some risk that
> >it might lead to many smaller disk writes which we might want to
> >control via some threshold (like we should not flush more than N
> >xacts). In this, we also need to ensure that the total memory freed
> >must be greater than the current change.
> >
> >I think we have some discussion around this point but didn't reach any
> >conclusion which means some more brainstorming is required.
> >
>
> I agree it's worth investigating, but I'm not sure it's necessary before
> committing v1 of the feature. I don't think there's a clear winner
> strategy, and the current approach works fairly well I think.
>
> The comment is concerned with the cost of ReorderBufferLargestTXN with
> many transactions, but we can only have certain number of top-level
> transactions (max_connections + certain number of not-yet-assigned
> subtransactions). And 0002 patch essentially gets rid of the subxacts
> entirely, further reducing the maximum number of xacts to walk.
>
That would be good, but I don't understand how. The second patch will
update the subxacts in top-level ReorderBufferTxn, but it won't remove
it from hash table. It also doesn't seem to be caring for considering
the size of subxacts in top-level xact, so not sure how will it reduce
the number of xacts to walk. I might be missing something here. Can
you explain a bit how 0002 patch would help in reducing the maximum
number of xacts to walk?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-27 09:03 Amit Kapila <amit.kapila16@gmail.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-27 09:03 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
>
> On 1/3/18 14:53, Tomas Vondra wrote:
> >> I don't see the need to tie this setting to maintenance_work_mem.
> >> maintenance_work_mem is often set to very large values, which could
> >> then have undesirable side effects on this use.
> >
> > Well, we need to pick some default value, and we can either use a fixed
> > value (not sure what would be a good default) or tie it to an existing
> > GUC. We only really have work_mem and maintenance_work_mem, and the
> > walsender process will never use more than one such buffer. Which seems
> > to be closer to maintenance_work_mem.
> >
> > Pretty much any default value can have undesirable side effects.
>
> Let's just make it an independent setting unless we know any better. We
> don't have a lot of settings that depend on other settings, and the ones
> we do have a very specific relationship.
>
> >> Moreover, the name logical_work_mem makes it sound like it's a logical
> >> version of work_mem. Maybe we could think of another name.
> >
> > I won't object to a better name, of course. Any proposals?
>
> logical_decoding_[work_]mem?
>
Having a separate variable for this can give more flexibility, but
OTOH it will add one more knob which user might not have a good idea
to set. What are the problems we see if directly use work_mem for
this case?
If we can't use work_mem, then I think the name proposed by you
(logical_decoding_work_mem) sounds good to me.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-27 09:40 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
3 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-09-27 09:40 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 11:37 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Wed, Sep 25, 2019 at 06:55:01PM +0530, Amit Kapila wrote:
> >On Tue, Sep 3, 2019 at 4:30 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
> >>
> >> In the interest of moving things forward, how far are we from making
> >> 0001 committable? If I understand correctly, the rest of this patchset
> >> depends on https://commitfest.postgresql.org/24/944/ which seems to be
> >> moving at a glacial pace (or, actually, slower, because glaciers do
> >> move, which cannot be said of that other patch.)
> >>
> >
> >I am not sure if it is completely correct that the other part of the
> >patch is dependent on that CF entry. I have studied both the threads
> >(not every detail) and it seems to me it is dependent on one of the
> >patches from that series which handles concurrent aborts. It is patch
> >0003-Gracefully-handle-concurrent-aborts-of-uncommitted-t.Jan4.patch
> >from what the Nikhil has posted on that thread [1]. Am, I wrong?
> >
>
> You're right - the part handling aborts is the only part required. There
> are dependencies on some other changes from the 2PC patch, but those are
> mostly refactorings that can be undone (e.g. switch from independent
> flags to a single bitmap in reorderbuffer).
>
> >So IIUC, the problem of concurrent aborts is that if we allow catalog
> >scans for in-progress transactions, then we might get wrong answers in
> >cases where somebody has performed Alter-Abort-Alter which is clearly
> >explained with an example in email [2]. To solve that problem Nikhil
> >seems to have written a patch [1] which detects these concurrent
> >aborts during a system table scan and then aborts the decoding of such
> >a transaction.
> >
> >Now, the problem is that patch has written considering 2PC
> >transactions and might not deal with all cases for in-progress
> >transactions especially when sub-transactions are involved as alluded
> >by Arseny Sher [3]. So, the problem seems to be for cases when some
> >sub-transaction aborts, but the main transaction still continued and
> >we try to decode it. Nikhil's patch won't be able to deal with it
> >because I think it just checks top-level xid whereas for this we need
> >to check all-subxids which I think is possible now as Tomas seems to
> >have written WAL for each xid-assignment. It might or might not be
> >the best solution to check the status of all-subxids, but I think
> >first we need to agree that the problem is just for concurrent aborts
> >and that we can solve it by using some part of the technology being
> >developed as part of patch "Logical decoding of two-phase
> >transactions" (https://commitfest.postgresql.org/24/944/) rather than
> >the entire patchset.
> >
> >I hope I am not saying something very obvious here and it helps in
> >moving this patch forward.
> >
>
> No, that's a good question, and I'm not sure what the answer is at the
> moment. My understanding was that the infrastructure in the 2PC patch is
> enough even for subtransactions, but I might be wrong.
>
I also think the patch that handles concurrent aborts should be
sufficient, but that need to be integrated with your patch. Earlier,
I thought we need to check whether any of the subtransaction is
aborted as mentioned by Arseny Sher, but now after thinking again
about that problem, it seems that checking only the status current
subtransaction should be sufficient. Because, if the user does
Rollback to Savepoint concurrently which aborts multiple
subtransactions, the latest one must be aborted as well which is what
I think we want to detect. Once we detect that we have two options
(a) restart the decode of that transaction by removing changes of all
subxacts or (b) somehow mark the transaction such that it gets decoded
only at the commit time.
>
> Maybe we should focus on the 0001 part for now - it can be committed
> indepently and does provide useful feature.
>
If that can be done sooner, then it is fine, but otherwise, preparing
the patches on top of HEAD can facilitate the review of those.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-27 11:25 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-27 11:25 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
>On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
><peter.eisentraut@2ndquadrant.com> wrote:
>>
>> On 1/3/18 14:53, Tomas Vondra wrote:
>> >> I don't see the need to tie this setting to maintenance_work_mem.
>> >> maintenance_work_mem is often set to very large values, which could
>> >> then have undesirable side effects on this use.
>> >
>> > Well, we need to pick some default value, and we can either use a fixed
>> > value (not sure what would be a good default) or tie it to an existing
>> > GUC. We only really have work_mem and maintenance_work_mem, and the
>> > walsender process will never use more than one such buffer. Which seems
>> > to be closer to maintenance_work_mem.
>> >
>> > Pretty much any default value can have undesirable side effects.
>>
>> Let's just make it an independent setting unless we know any better. We
>> don't have a lot of settings that depend on other settings, and the ones
>> we do have a very specific relationship.
>>
>> >> Moreover, the name logical_work_mem makes it sound like it's a logical
>> >> version of work_mem. Maybe we could think of another name.
>> >
>> > I won't object to a better name, of course. Any proposals?
>>
>> logical_decoding_[work_]mem?
>>
>
>Having a separate variable for this can give more flexibility, but
>OTOH it will add one more knob which user might not have a good idea
>to set. What are the problems we see if directly use work_mem for
>this case?
>
IMHO it's similar to autovacuum_work_mem - we have an independent
setting, but most people use it as -1 so we use maintenance_work_mem as
a default value. I think it makes sense to do the same thing here.
It does ad an extra knob anyway (I don't think we should just use
maintenance_work_mem directly, the user should have an option to
override it when needed). But most users will not notice.
FWIW I don't think we should use work_mem, maintenace_work_mem seems
somewhat more appropriate here (not related to queries, etc.).
>If we can't use work_mem, then I think the name proposed by you
>(logical_decoding_work_mem) sounds good to me.
>
Yes, that name seems better.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-28 08:06 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-28 08:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
> >On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
> ><peter.eisentraut@2ndquadrant.com> wrote:
> >>
> >> On 1/3/18 14:53, Tomas Vondra wrote:
> >> >> I don't see the need to tie this setting to maintenance_work_mem.
> >> >> maintenance_work_mem is often set to very large values, which could
> >> >> then have undesirable side effects on this use.
> >> >
> >> > Well, we need to pick some default value, and we can either use a fixed
> >> > value (not sure what would be a good default) or tie it to an existing
> >> > GUC. We only really have work_mem and maintenance_work_mem, and the
> >> > walsender process will never use more than one such buffer. Which seems
> >> > to be closer to maintenance_work_mem.
> >> >
> >> > Pretty much any default value can have undesirable side effects.
> >>
> >> Let's just make it an independent setting unless we know any better. We
> >> don't have a lot of settings that depend on other settings, and the ones
> >> we do have a very specific relationship.
> >>
> >> >> Moreover, the name logical_work_mem makes it sound like it's a logical
> >> >> version of work_mem. Maybe we could think of another name.
> >> >
> >> > I won't object to a better name, of course. Any proposals?
> >>
> >> logical_decoding_[work_]mem?
> >>
> >
> >Having a separate variable for this can give more flexibility, but
> >OTOH it will add one more knob which user might not have a good idea
> >to set. What are the problems we see if directly use work_mem for
> >this case?
> >
>
> IMHO it's similar to autovacuum_work_mem - we have an independent
> setting, but most people use it as -1 so we use maintenance_work_mem as
> a default value. I think it makes sense to do the same thing here.
>
> It does ad an extra knob anyway (I don't think we should just use
> maintenance_work_mem directly, the user should have an option to
> override it when needed). But most users will not notice.
>
> FWIW I don't think we should use work_mem, maintenace_work_mem seems
> somewhat more appropriate here (not related to queries, etc.).
>
I have the same concern for using maintenace_work_mem as Peter E.
which is that the value of maintenace_work_mem will generally be
higher which is suitable for its current purpose, but not for the
purpose this patch is using. AFAIU, at this stage we want a better
memory accounting system for logical decoding and we are not sure what
is a good value for this variable. So, I think using work_mem or
maintenace_work_mem should serve the purpose. Later, if we have
requirements from people to have better control over the memory
required for this purpose then we can introduce a new variable.
I understand that currently work_mem is primarily tied with memory
used for query workspaces, but it might be okay to extend it for this
purpose. Another point is that the default for that sound to be more
appealing for this case. I can see the argument against it which is
having a separate variable will make the things look clean and give
better control. So, if we can't convince ourselves for using
work_mem, we can introduce a new guc variable and keep the default as
4MB or work_mem.
I feel it is always tempting to introduce a new guc for the different
tasks unless there is an exact match, but OTOH, having lesser guc's
has its own advantage which is that people don't have to bother about
a new setting which they need to tune and especially for which they
can't decide with ease. I am not telling that we should not introduce
new guc when it is required, but just to give more thought before
doing so.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-28 12:03 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
3 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-09-28 12:03 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 11:37 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> Hi,
>
> Attached is an updated patch series, rebased on current master. It does
> fix one memory accounting bug in ReorderBufferToastReplace (the code was
> not properly updating the amount of memory).
>
Few comments on 0001
1.
I am getting below linking error in pgoutput when compiling the patch
on my windows system:
pgoutput.obj : error LNK2001: unresolved external symbol _logical_work_mem
You need to use PGDLLIMPORT for logical_work_mem.
2. After, I fixed above and tried some basic test, it fails with below
callstack:
postgres.exe!ExceptionalCondition(const char *
conditionName=0x00d92854, const char * errorType=0x00d928bc, const
char * fileName=0x00d92e60,
int lineNumber=2148) Line 55
postgres.exe!ReorderBufferChangeMemoryUpdate(ReorderBuffer *
rb=0x02693390, ReorderBufferChange * change=0x0269dd38, bool
addition=true) Line 2148
postgres.exe!ReorderBufferQueueChange(ReorderBuffer * rb=0x02693390,
unsigned int xid=525, unsigned __int64 lsn=36083720,
ReorderBufferChange
* change=0x0269dd38) Line 635
postgres.exe!DecodeInsert(LogicalDecodingContext * ctx=0x0268ef80,
XLogRecordBuffer * buf=0x012cf718) Line 716 + 0x24 bytes C
postgres.exe!DecodeHeapOp(LogicalDecodingContext * ctx=0x0268ef80,
XLogRecordBuffer * buf=0x012cf718) Line 437 + 0xd bytes C
postgres.exe!LogicalDecodingProcessRecord(LogicalDecodingContext *
ctx=0x0268ef80, XLogReaderState * record=0x0268f228) Line 129
postgres.exe!pg_logical_slot_get_changes_guts(FunctionCallInfoBaseData
* fcinfo=0x02688680, bool confirm=true, bool binary=false) Line 307
postgres.exe!pg_logical_slot_get_changes(FunctionCallInfoBaseData *
fcinfo=0x02688680) Line 376
Basically, the assert added by you in ReorderBufferChangeMemoryUpdate
failed. Then, I explored a bit and it seems that you have missed
assigning a value to txn, a new variable added by this patch in
structure ReorderBufferChange:
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
3.
@@ -206,6 +206,17 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_work_mem</varname>.
+ </para>
+ </listitem>
+ </varlistentry>
I don't see any explanation of how this will be useful? How can a
subscriber predict the amount of memory required by a publisher for
decoding? This is more unpredictable because when initially the
changes are recorded in ReorderBuffer, it doesn't even filter
corresponding to any publisher. Do we really need this? I think
giving more knobs to the user is helpful when they can someway know
how to use it. In this case, it is not clear whether the user can
ever use this.
4. Can we some way expose the memory consumed by ReorderBuffer? If
so, we might be able to write some tests covering new functionality.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-28 19:09 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-28 19:09 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Sep 28, 2019 at 01:36:46PM +0530, Amit Kapila wrote:
>On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
>> >On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
>> ><peter.eisentraut@2ndquadrant.com> wrote:
>> >>
>> >> On 1/3/18 14:53, Tomas Vondra wrote:
>> >> >> I don't see the need to tie this setting to maintenance_work_mem.
>> >> >> maintenance_work_mem is often set to very large values, which could
>> >> >> then have undesirable side effects on this use.
>> >> >
>> >> > Well, we need to pick some default value, and we can either use a fixed
>> >> > value (not sure what would be a good default) or tie it to an existing
>> >> > GUC. We only really have work_mem and maintenance_work_mem, and the
>> >> > walsender process will never use more than one such buffer. Which seems
>> >> > to be closer to maintenance_work_mem.
>> >> >
>> >> > Pretty much any default value can have undesirable side effects.
>> >>
>> >> Let's just make it an independent setting unless we know any better. We
>> >> don't have a lot of settings that depend on other settings, and the ones
>> >> we do have a very specific relationship.
>> >>
>> >> >> Moreover, the name logical_work_mem makes it sound like it's a logical
>> >> >> version of work_mem. Maybe we could think of another name.
>> >> >
>> >> > I won't object to a better name, of course. Any proposals?
>> >>
>> >> logical_decoding_[work_]mem?
>> >>
>> >
>> >Having a separate variable for this can give more flexibility, but
>> >OTOH it will add one more knob which user might not have a good idea
>> >to set. What are the problems we see if directly use work_mem for
>> >this case?
>> >
>>
>> IMHO it's similar to autovacuum_work_mem - we have an independent
>> setting, but most people use it as -1 so we use maintenance_work_mem as
>> a default value. I think it makes sense to do the same thing here.
>>
>> It does ad an extra knob anyway (I don't think we should just use
>> maintenance_work_mem directly, the user should have an option to
>> override it when needed). But most users will not notice.
>>
>> FWIW I don't think we should use work_mem, maintenace_work_mem seems
>> somewhat more appropriate here (not related to queries, etc.).
>>
>
>I have the same concern for using maintenace_work_mem as Peter E.
>which is that the value of maintenace_work_mem will generally be
>higher which is suitable for its current purpose, but not for the
>purpose this patch is using. AFAIU, at this stage we want a better
>memory accounting system for logical decoding and we are not sure what
>is a good value for this variable. So, I think using work_mem or
>maintenace_work_mem should serve the purpose. Later, if we have
>requirements from people to have better control over the memory
>required for this purpose then we can introduce a new variable.
>
>I understand that currently work_mem is primarily tied with memory
>used for query workspaces, but it might be okay to extend it for this
>purpose. Another point is that the default for that sound to be more
>appealing for this case. I can see the argument against it which is
>having a separate variable will make the things look clean and give
>better control. So, if we can't convince ourselves for using
>work_mem, we can introduce a new guc variable and keep the default as
>4MB or work_mem.
>
>I feel it is always tempting to introduce a new guc for the different
>tasks unless there is an exact match, but OTOH, having lesser guc's
>has its own advantage which is that people don't have to bother about
>a new setting which they need to tune and especially for which they
>can't decide with ease. I am not telling that we should not introduce
>new guc when it is required, but just to give more thought before
>doing so.
>
I do think having a separate GUC is a must, irrespectedly of what other
GUC (if any) is used as a default. You're right the maintenance_work_mem
value might be too high (e.g. in cases with many subscriptions), but the
same issue applies to work_mem - there's no guarantee work_mem is lower
than maintenance_work_mem, and in analytics databases it may be set very
high. So work_mem does not really solve the issue
IMHO we can't really do without a new GUC. It's not difficult to create
examples that would benefit from small/large memory limit, depending on
the number of subscriptions etc.
I do however agree the GUC does not have to be tied to any existing one,
it was just an attempt to use a more sensible default value. I do think
m_w_m would be fine, but I can live with using an explicit value.
So that's what I did in the attached patch - I've renamed the GUC to
logical_decoding_work_mem, detached it from m_w_m and set the default to
64MB (i.e. the same default as m_w_m). It should also fix all the issues
from the recent reviews (at least I believe so).
I've realized that one of the subsequent patches allows overriding the
limit for individual subscriptions (in the CREATE SUBSCRIPTION command).
I think it'd be good to move this bit forward, but I think it can be
done in a separate patch.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch.gz (application/gzip, 10.2K) download
0002-Immediately-WAL-log-assignments.patch.gz (application/gzip, 6.6K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 6.1K) download
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch.gz (application/gzip, 2.5K) download
0006-Support-decoding-of-two-phase-transactions-at-PREPAR.patch.gz (application/gzip, 9.0K) download
0007-Gracefully-handle-concurrent-aborts-of-uncommitted.patch.gz (application/gzip, 4.4K) download
0008-Teach-test_decoding-plugin-to-work-with-2PC.patch.gz (application/gzip, 5.8K) download
0009-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 12.6K) download
0010-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 19.8K) download
0011-Track-statistics-for-streaming-spilling.patch.gz (application/gzip, 4.2K) download
0012-Enable-streaming-for-all-subscription-TAP-tests.patch.gz (application/gzip, 2.3K) download
0013-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch.gz (application/gzip, 628B) download
0014-Add-TAP-test-for-streaming-vs.-DDL.patch.gz (application/gzip, 1.5K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-29 04:34 Dilip Kumar <dilipbalaut@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
3 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-09-29 04:34 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Thu, Sep 26, 2019 at 11:38 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> No, that's a good question, and I'm not sure what the answer is at the
> moment. My understanding was that the infrastructure in the 2PC patch is
> enough even for subtransactions, but I might be wrong. I need to think
> about that for a while.
>
IIUC, for 2PC it's enough to check whether the main transaction is
aborted or not but for the in-progress transaction it's possible that
the current subtransaction might have done catalog changes and it
might get aborted when we are decoding. So we need to extend an
infrastructure such that we can check the status of the transaction
for which we are decoding the change. Also, I think we need to handle
the ERRCODE_TRANSACTION_ROLLBACK and ignore it.
I have attached a small patch to handle this which can be applied on
top of your patch set.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
handle_concurrent_abort_for_in_progress_transaction.patch (application/octet-stream, 1.7K) download
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 163395f..ca7e2d7 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3449,6 +3449,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3579,6 +3580,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3891,6 +3899,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3909,7 +3920,14 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ FlushErrorState();
}
PG_END_TRY();
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-29 05:54 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-09-29 05:54 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Sat, Sep 28, 2019 at 01:36:46PM +0530, Amit Kapila wrote:
> >On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
>
> I do think having a separate GUC is a must, irrespectedly of what other
> GUC (if any) is used as a default. You're right the maintenance_work_mem
> value might be too high (e.g. in cases with many subscriptions), but the
> same issue applies to work_mem - there's no guarantee work_mem is lower
> than maintenance_work_mem, and in analytics databases it may be set very
> high. So work_mem does not really solve the issue
>
> IMHO we can't really do without a new GUC. It's not difficult to create
> examples that would benefit from small/large memory limit, depending on
> the number of subscriptions etc.
>
> I do however agree the GUC does not have to be tied to any existing one,
> it was just an attempt to use a more sensible default value. I do think
> m_w_m would be fine, but I can live with using an explicit value.
>
> So that's what I did in the attached patch - I've renamed the GUC to
> logical_decoding_work_mem, detached it from m_w_m and set the default to
> 64MB (i.e. the same default as m_w_m).
Fair enough, let's not argue more on this unless someone else wants to
share his opinion.
> It should also fix all the issues
> from the recent reviews (at least I believe so).
>
Have you given any thought on creating a test case for this patch? I
think you also told that you will test some worst-case scenarios and
report the numbers so that we are convinced that the current eviction
algorithm is good.
> I've realized that one of the subsequent patches allows overriding the
> limit for individual subscriptions (in the CREATE SUBSCRIPTION command).
> I think it'd be good to move this bit forward, but I think it can be
> done in a separate patch.
>
Yeah, it is better to deal it separately as I am also not entirely
convinced at this stage about this parameter. I have mentioned the
same in the previous email as well.
While glancing through the changes, I noticed a small thing:
+#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
I guess this need to be updated.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-29 17:30 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2019-09-29 17:30 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2019-Sep-29, Amit Kapila wrote:
> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
> > So that's what I did in the attached patch - I've renamed the GUC to
> > logical_decoding_work_mem, detached it from m_w_m and set the default to
> > 64MB (i.e. the same default as m_w_m).
>
> Fair enough, let's not argue more on this unless someone else wants to
> share his opinion.
I just read this part of the conversation and I agree that having a
separate GUC with its own value independent from other GUCs is a good
solution. Tying it to m_w_m seemed reasonable, but it's true that
people frequently set m_w_m very high, and it would be undesirable to
propagate that value to logical decoding memory usage.
I wonder what would constitute good advice on how to set this value, I
mean what is the metric that the user needs to be thinking about. Is
it the total of memory required to keep all concurrent write transactions
in memory? (Quick example: if you do 2048 wTPS and each transaction
lasts 1s, and each transaction does 1kB of logically-decoded changes,
then ~2MB are sufficient for the average case. Is that correct? I
*think* that full-page images do not count, correct? With these things
in mind users could go through pg_waldump output and figure out what to
set the value to.)
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-09-29 17:56 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-29 17:56 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 02:30:44PM -0300, Alvaro Herrera wrote:
>On 2019-Sep-29, Amit Kapila wrote:
>
>> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>
>> > So that's what I did in the attached patch - I've renamed the GUC to
>> > logical_decoding_work_mem, detached it from m_w_m and set the default to
>> > 64MB (i.e. the same default as m_w_m).
>>
>> Fair enough, let's not argue more on this unless someone else wants to
>> share his opinion.
>
>I just read this part of the conversation and I agree that having a
>separate GUC with its own value independent from other GUCs is a good
>solution. Tying it to m_w_m seemed reasonable, but it's true that
>people frequently set m_w_m very high, and it would be undesirable to
>propagate that value to logical decoding memory usage.
>
>
>I wonder what would constitute good advice on how to set this value, I
>mean what is the metric that the user needs to be thinking about. Is
>it the total of memory required to keep all concurrent write transactions
>in memory? (Quick example: if you do 2048 wTPS and each transaction
>lasts 1s, and each transaction does 1kB of logically-decoded changes,
>then ~2MB are sufficient for the average case. Is that correct?
Yes, something like that. Essentially we'd like to keep all concurrent
transactions decoded in memory, to eliminate the need to spill to disk.
One of the subsequent patches adds some subscription-level stats, so
maybe we don't need to worry about this too much - the stats seem like a
better source of information for tuning.
>I *think* that full-page images do not count, correct? With these
>things in mind users could go through pg_waldump output and figure out
>what to set the value to.)
>
Right, FPW do not matter here.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-01 13:25 Amit Kapila <amit.kapila16@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-01 13:25 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila <amit.kapila16@gmail.com>
wrote:
> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
>
> Yeah, it is better to deal it separately as I am also not entirely
> convinced at this stage about this parameter. I have mentioned the
> same in the previous email as well.
>
> While glancing through the changes, I noticed a small thing:
> +#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
maintenance_work_mem
>
> I guess this need to be updated.
>
On further testing, I found that the patch seems to have problems with
toast. Consider below scenario:
Session-1
Create table large_text(t1 text);
INSERT INTO large_text
SELECT (SELECT string_agg('x', ',')
FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
Session-2
SELECT * FROM pg_create_logical_replication_slot('regression_slot',
'test_decoding');
SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
*--kaboom*
The second statement in Session-2 leads to a crash.
Other than that, I am not sure if the changes related to spill to disk
after logical_decoding_work_mem works for toast table as I couldn't hit
that code for toast table case, but I might be missing something. As
mentioned previously, I feel there should be some way to test whether this
patch works for the cases it claims to work. As of now, I have to check
via debugging. Let me know if there is any way, I can test this.
I am reluctant to say, but I think this patch still needs some more work
(review, test, rework) before we can commit it.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-01 13:51 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-10-01 13:51 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
>On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila <amit.kapila16@gmail.com>
>wrote:
>> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> >
>>
>> Yeah, it is better to deal it separately as I am also not entirely
>> convinced at this stage about this parameter. I have mentioned the
>> same in the previous email as well.
>>
>> While glancing through the changes, I noticed a small thing:
>> +#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
>maintenance_work_mem
>>
>> I guess this need to be updated.
>>
>
>On further testing, I found that the patch seems to have problems with
>toast. Consider below scenario:
>Session-1
>Create table large_text(t1 text);
>INSERT INTO large_text
>SELECT (SELECT string_agg('x', ',')
>FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>
>Session-2
>SELECT * FROM pg_create_logical_replication_slot('regression_slot',
>'test_decoding');
>SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
>*--kaboom*
>
>The second statement in Session-2 leads to a crash.
>
OK, thanks for the report - will investigate.
>Other than that, I am not sure if the changes related to spill to disk
>after logical_decoding_work_mem works for toast table as I couldn't hit
>that code for toast table case, but I might be missing something. As
>mentioned previously, I feel there should be some way to test whether this
>patch works for the cases it claims to work. As of now, I have to check
>via debugging. Let me know if there is any way, I can test this.
>
That's one of the reasons why I proposed to move the statistics (which
say how many transactions / bytes were spilled to disk) from a later
patch in the series. I don't think there's a better way.
>I am reluctant to say, but I think this patch still needs some more work
>(review, test, rework) before we can commit it.
>
I agreee.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-01 22:57 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-01 22:57 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >
> >On further testing, I found that the patch seems to have problems with
> >toast. Consider below scenario:
> >Session-1
> >Create table large_text(t1 text);
> >INSERT INTO large_text
> >SELECT (SELECT string_agg('x', ',')
> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >
> >Session-2
> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >'test_decoding');
> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
> >*--kaboom*
> >
> >The second statement in Session-2 leads to a crash.
> >
>
> OK, thanks for the report - will investigate.
>
It was an assertion failure in ReorderBufferCleanupTXN at below line:
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
> >Other than that, I am not sure if the changes related to spill to disk
> >after logical_decoding_work_mem works for toast table as I couldn't hit
> >that code for toast table case, but I might be missing something. As
> >mentioned previously, I feel there should be some way to test whether this
> >patch works for the cases it claims to work. As of now, I have to check
> >via debugging. Let me know if there is any way, I can test this.
> >
>
> That's one of the reasons why I proposed to move the statistics (which
> say how many transactions / bytes were spilled to disk) from a later
> patch in the series. I don't think there's a better way.
>
>
I like that idea, but I think you need to split that patch to only get the
stats related to the spill. It would be easier to review if you can
prepare that atop of
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-02 22:32 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 3 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-10-02 22:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
>On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
>wrote:
>
>> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
>> >
>> >On further testing, I found that the patch seems to have problems with
>> >toast. Consider below scenario:
>> >Session-1
>> >Create table large_text(t1 text);
>> >INSERT INTO large_text
>> >SELECT (SELECT string_agg('x', ',')
>> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>> >
>> >Session-2
>> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
>> >'test_decoding');
>> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
>> >*--kaboom*
>> >
>> >The second statement in Session-2 leads to a crash.
>> >
>>
>> OK, thanks for the report - will investigate.
>>
>
>It was an assertion failure in ReorderBufferCleanupTXN at below line:
>+ /* Check we're not mixing changes from different transactions. */
>+ Assert(change->txn == txn);
>
Can you still reproduce this issue with the patch I sent on 28/9? I have
been unable to trigger the failure, and it seems pretty similar to the
failure you reported (and I fixed) on 28/9.
>> >Other than that, I am not sure if the changes related to spill to disk
>> >after logical_decoding_work_mem works for toast table as I couldn't hit
>> >that code for toast table case, but I might be missing something. As
>> >mentioned previously, I feel there should be some way to test whether this
>> >patch works for the cases it claims to work. As of now, I have to check
>> >via debugging. Let me know if there is any way, I can test this.
>> >
>>
>> That's one of the reasons why I proposed to move the statistics (which
>> say how many transactions / bytes were spilled to disk) from a later
>> patch in the series. I don't think there's a better way.
>>
>>
>I like that idea, but I think you need to split that patch to only get the
>stats related to the spill. It would be easier to review if you can
>prepare that atop of
>0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
>
Sure, I wasn't really proposing to adding all stats from that patch,
including those related to streaming. We need to extract just those
related to spilling. And yes, it needs to be moved right after 0001.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-03 07:48 Dilip Kumar <dilipbalaut@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 3 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-03 07:48 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
I have attempted to test the performance of (Stream + Spill) vs
(Stream + BGW pool) and I can see the similar gain what Alexey had
shown[1].
In addition to this, I have rebased the latest patchset [2] without
the two-phase logical decoding patch set.
Test results:
I have repeated the same test as Alexy[1] for 1kk and 1kk data and
here is my result
Stream + Spill
N time on master(sec) Total xact time (sec)
1kk 6 21
3kk 18 55
Stream + BGW pool
N time on master(sec) Total xact time (sec)
1kk 6 13
3kk 19 35
Patch details:
All the patches are the same as posted on [2] except
1. 0006-Gracefully-handle-concurrent-aborts-of-uncommitted -> I have
removed the handling of error which is specific for 2PC
2. 0007-Implement-streaming-mode-in-ReorderBuffer -> Rebased without 2PC
3. 0009-Extend-the-concurrent-abort-handling-for-in-progress -> New
patch to handle concurrent abort error for the in-progress transaction
and also add handling for the sub transaction's abort.
4. v3-0014-BGWorkers-pool-for-streamed-transactions-apply -> Rebased
Alexey's patch
[1] https://www.postgresql.org/message-id/8eda5118-2dd0-79a1-4fe9-eec7e334de17%40postgrespro.ru
[2] https://www.postgresql.org/message-id/20190928190917.hrpknmq76v3ts3lj%40development
On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
> >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
> >wrote:
> >
> >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >> >
> >> >On further testing, I found that the patch seems to have problems with
> >> >toast. Consider below scenario:
> >> >Session-1
> >> >Create table large_text(t1 text);
> >> >INSERT INTO large_text
> >> >SELECT (SELECT string_agg('x', ',')
> >> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >> >
> >> >Session-2
> >> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >> >'test_decoding');
> >> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
> >> >*--kaboom*
> >> >
> >> >The second statement in Session-2 leads to a crash.
> >> >
> >>
> >> OK, thanks for the report - will investigate.
> >>
> >
> >It was an assertion failure in ReorderBufferCleanupTXN at below line:
> >+ /* Check we're not mixing changes from different transactions. */
> >+ Assert(change->txn == txn);
> >
>
> Can you still reproduce this issue with the patch I sent on 28/9? I have
> been unable to trigger the failure, and it seems pretty similar to the
> failure you reported (and I fixed) on 28/9.
>
> >> >Other than that, I am not sure if the changes related to spill to disk
> >> >after logical_decoding_work_mem works for toast table as I couldn't hit
> >> >that code for toast table case, but I might be missing something. As
> >> >mentioned previously, I feel there should be some way to test whether this
> >> >patch works for the cases it claims to work. As of now, I have to check
> >> >via debugging. Let me know if there is any way, I can test this.
> >> >
> >>
> >> That's one of the reasons why I proposed to move the statistics (which
> >> say how many transactions / bytes were spilled to disk) from a later
> >> patch in the series. I don't think there's a better way.
> >>
> >>
> >I like that idea, but I think you need to split that patch to only get the
> >stats related to the spill. It would be easier to review if you can
> >prepare that atop of
> >0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
> >
>
> Sure, I wasn't really proposing to adding all stats from that patch,
> including those related to streaming. We need to extract just those
> related to spilling. And yes, it needs to be moved right after 0001.
>
> regards
>
> --
> Tomas Vondra http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch (application/octet-stream, 33.6K) download
From d2830f660ca86dd84b5c377748d133cfe57463ae Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:04:54 +0200
Subject: [PATCH 01/13] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to trigger very aggressive streaming. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem may be set either in postgresql.conf,
in which case it serves as the default for all publishers on that
instance, or when creating the subscription, using a work_mem
paramemter in the WITH clause (specifies number of kilobytes).
---
doc/src/sgml/config.sgml | 21 ++
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/reorderbuffer.c | 292 ++++++++++++++++++++-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++-
src/backend/utils/misc/guc.c | 36 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/reorderbuffer.h | 16 ++
src/include/replication/walreceiver.h | 1 +
13 files changed, 441 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 619ac8c..a207ac2 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1715,6 +1715,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee283..7e3ba8e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e67a58..d85e831 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +98,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -182,6 +185,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +338,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -341,7 +356,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -419,6 +434,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -682,10 +703,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -710,6 +734,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -721,7 +752,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -759,7 +791,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -796,7 +828,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a..65b3266 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8ce28ad..6228140 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is then
+ * serialized to disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when
+ * reaching memory limit. So we just don't count them, because it
+ * might easily trigger a pointless attempt to spill/stream.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2230,20 +2323,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2670,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3019,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters).
+ * And we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3248,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3437,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 11e6331..f737afb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1725,6 +1725,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757..317c5d4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/inval.h"
#include "utils/int8.h"
#include "utils/memutils.h"
@@ -90,11 +91,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -140,6 +142,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +199,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 2178e1c..5d7e687 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -65,6 +65,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
+#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
@@ -191,6 +192,7 @@ static bool check_maxconnections(int *newval, void **extra, GucSource source);
static bool check_max_worker_processes(int *newval, void **extra, GucSource source);
static bool check_autovacuum_max_workers(int *newval, void **extra, GucSource source);
static bool check_max_wal_senders(int *newval, void **extra, GucSource source);
+static bool check_logical_decoding_work_mem(int *newval, void **extra, GucSource source);
static bool check_autovacuum_work_mem(int *newval, void **extra, GucSource source);
static bool check_effective_io_concurrency(int *newval, void **extra, GucSource source);
static void assign_effective_io_concurrency(int newval, void *extra);
@@ -2251,6 +2253,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk or streaming."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ -1, -1, MAX_KILOBYTES,
+ check_logical_decoding_work_mem, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -11286,6 +11300,28 @@ check_max_wal_senders(int *newval, void **extra, GucSource source)
}
static bool
+check_logical_decoding_work_mem(int *newval, void **extra, GucSource source)
+{
+ /*
+ * -1 indicates fallback.
+ *
+ * If we haven't yet changed the boot_val default of -1, just let it be.
+ * logical decoding will look to maintenance_work_mem instead.
+ */
+ if (*newval == -1)
+ return true;
+
+ /*
+ * We clamp manually-set values to at least 64kB. The maintenance_work_mem
+ * uses a higher minimum value (1MB), so this is OK.
+ */
+ if (*newval < 64)
+ *newval = 64;
+
+ return true;
+}
+
+static bool
check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
{
/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3..00a22b8 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..4dcef80 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
0002-Immediately-WAL-log-assignments.patch (application/octet-stream, 19.1K) download
From f70b1e79f8dcb23f1acffd5a29758e7952e61d9c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:07:31 +0200
Subject: [PATCH 02/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/rmgrdesc/xactdesc.c | 26 ------
src/backend/access/transam/xact.c | 152 +++++++++----------------------
src/backend/access/transam/xlog.c | 2 -
src/backend/access/transam/xloginsert.c | 22 ++++-
src/backend/access/transam/xlogreader.c | 5 +
src/backend/replication/logical/decode.c | 39 ++++----
src/include/access/xact.h | 15 +--
src/include/access/xlog.h | 2 +
src/include/access/xlogreader.h | 3 +
src/include/access/xlogrecord.h | 1 +
src/tools/pgindent/typedefs.list | 1 -
11 files changed, 96 insertions(+), 172 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index a61f38d..66fc8fb 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -293,17 +293,6 @@ xact_desc_abort(StringInfo buf, uint8 info, xl_xact_abort *xlrec)
}
}
-static void
-xact_desc_assignment(StringInfo buf, xl_xact_assignment *xlrec)
-{
- int i;
-
- appendStringInfoString(buf, "subxacts:");
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- appendStringInfo(buf, " %u", xlrec->xsub[i]);
-}
-
void
xact_desc(StringInfo buf, XLogReaderState *record)
{
@@ -323,18 +312,6 @@ xact_desc(StringInfo buf, XLogReaderState *record)
xact_desc_abort(buf, XLogRecGetInfo(record), xlrec);
}
- else if (info == XLOG_XACT_ASSIGNMENT)
- {
- xl_xact_assignment *xlrec = (xl_xact_assignment *) rec;
-
- /*
- * Note that we ignore the WAL record's xid, since we're more
- * interested in the top-level xid that issued the record and which
- * xids are being reported here.
- */
- appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
- xact_desc_assignment(buf, xlrec);
- }
}
const char *
@@ -359,9 +336,6 @@ xact_identify(uint8 info)
case XLOG_XACT_ABORT_PREPARED:
id = "ABORT_PREPARED";
break;
- case XLOG_XACT_ASSIGNMENT:
- id = "ASSIGNMENT";
- break;
}
return id;
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9162286..33141fb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -188,9 +188,9 @@ typedef struct TransactionStateData
int prevSecContext; /* previous SecurityRestrictionContext */
bool prevXactReadOnly; /* entry-time xact r/o state */
bool startedInRecovery; /* did we start in recovery? */
- bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -225,13 +225,6 @@ static TransactionStateData TopTransactionStateData = {
.blockState = TBLOCK_DEFAULT,
};
-/*
- * unreportedXids holds XIDs of all subtransactions that have not yet been
- * reported in an XLOG_XACT_ASSIGNMENT record.
- */
-static int nUnreportedXids;
-static TransactionId unreportedXids[PGPROC_MAX_CACHED_SUBXIDS];
-
static TransactionState CurrentTransactionState = &TopTransactionStateData;
/*
@@ -502,19 +495,6 @@ GetCurrentFullTransactionIdIfAny(void)
}
/*
- * MarkCurrentTransactionIdLoggedIfAny
- *
- * Remember that the current xid - if it is assigned - now has been wal logged.
- */
-void
-MarkCurrentTransactionIdLoggedIfAny(void)
-{
- if (FullTransactionIdIsValid(CurrentTransactionState->fullTransactionId))
- CurrentTransactionState->didLogXid = true;
-}
-
-
-/*
* GetStableLatestTransactionId
*
* Get the transaction's XID if it has one, else read the next-to-be-assigned
@@ -555,7 +535,6 @@ AssignTransactionId(TransactionState s)
{
bool isSubXact = (s->parent != NULL);
ResourceOwner currentOwner;
- bool log_unknown_top = false;
/* Assert that caller didn't screw up */
Assert(!FullTransactionIdIsValid(s->fullTransactionId));
@@ -598,20 +577,6 @@ AssignTransactionId(TransactionState s)
}
/*
- * When wal_level=logical, guarantee that a subtransaction's xid can only
- * be seen in the WAL stream if its toplevel xid has been logged before.
- * If necessary we log an xact_assignment record with fewer than
- * PGPROC_MAX_CACHED_SUBXIDS. Note that it is fine if didLogXid isn't set
- * for a transaction even though it appears in a WAL record, we just might
- * superfluously log something. That can happen when an xid is included
- * somewhere inside a wal record, but not in XLogRecord->xl_xid, like in
- * xl_standby_locks.
- */
- if (isSubXact && XLogLogicalInfoActive() &&
- !TopTransactionStateData.didLogXid)
- log_unknown_top = true;
-
- /*
* Generate a new FullTransactionId and record its xid in PG_PROC and
* pg_subtrans.
*
@@ -646,59 +611,6 @@ AssignTransactionId(TransactionState s)
XactLockTableInsert(XidFromFullTransactionId(s->fullTransactionId));
CurrentResourceOwner = currentOwner;
-
- /*
- * Every PGPROC_MAX_CACHED_SUBXIDS assigned transaction ids within each
- * top-level transaction we issue a WAL record for the assignment. We
- * include the top-level xid and all the subxids that have not yet been
- * reported using XLOG_XACT_ASSIGNMENT records.
- *
- * This is required to limit the amount of shared memory required in a hot
- * standby server to keep track of in-progress XIDs. See notes for
- * RecordKnownAssignedTransactionIds().
- *
- * We don't keep track of the immediate parent of each subxid, only the
- * top-level transaction that each subxact belongs to. This is correct in
- * recovery only because aborted subtransactions are separately WAL
- * logged.
- *
- * This is correct even for the case where several levels above us didn't
- * have an xid assigned as we recursed up to them beforehand.
- */
- if (isSubXact && XLogStandbyInfoActive())
- {
- unreportedXids[nUnreportedXids] = XidFromFullTransactionId(s->fullTransactionId);
- nUnreportedXids++;
-
- /*
- * ensure this test matches similar one in
- * RecoverPreparedTransactions()
- */
- if (nUnreportedXids >= PGPROC_MAX_CACHED_SUBXIDS ||
- log_unknown_top)
- {
- xl_xact_assignment xlrec;
-
- /*
- * xtop is always set by now because we recurse up transaction
- * stack to the highest unassigned xid and then come back down
- */
- xlrec.xtop = GetTopTransactionId();
- Assert(TransactionIdIsValid(xlrec.xtop));
- xlrec.nsubxacts = nUnreportedXids;
-
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, MinSizeOfXactAssignment);
- XLogRegisterData((char *) unreportedXids,
- nUnreportedXids * sizeof(TransactionId));
-
- (void) XLogInsert(RM_XACT_ID, XLOG_XACT_ASSIGNMENT);
-
- nUnreportedXids = 0;
- /* mark top, not current xact as having been logged */
- TopTransactionStateData.didLogXid = true;
- }
- }
}
/*
@@ -1792,13 +1704,6 @@ AtSubAbort_childXids(void)
s->childXids = NULL;
s->nChildXids = 0;
s->maxChildXids = 0;
-
- /*
- * We could prune the unreportedXids array here. But we don't bother. That
- * would potentially reduce number of XLOG_XACT_ASSIGNMENT records but it
- * would likely introduce more CPU time into the more common paths, so we
- * choose not to do that.
- */
}
/* ----------------------------------------------------------------
@@ -1963,12 +1868,6 @@ StartTransaction(void)
currentCommandIdUsed = false;
/*
- * initialize reported xid accounting
- */
- nUnreportedXids = 0;
- s->didLogXid = false;
-
- /*
* must initialize resource-management stuff first
*/
AtStart_Memory();
@@ -5095,6 +4994,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -5990,14 +5890,46 @@ xact_redo(XLogReaderState *record)
XLogRecGetOrigin(record));
LWLockRelease(TwoPhaseStateLock);
}
- else if (info == XLOG_XACT_ASSIGNMENT)
- {
- xl_xact_assignment *xlrec = (xl_xact_assignment *) XLogRecGetData(record);
-
- if (standbyState >= STANDBY_INITIALIZED)
- ProcArrayApplyXidAssignment(xlrec->xtop,
- xlrec->nsubxacts, xlrec->xsub);
- }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 790e2c8..b1daa05 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1121,8 +1121,6 @@ XLogInsertRecord(XLogRecData *rdata,
*/
WALInsertLockRelease();
- MarkCurrentTransactionIdLoggedIfAny();
-
END_CRIT_SECTION();
/*
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 3ec67d4..15ce79c 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c8b0d23..3b02fbf 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1072,6 +1072,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1110,6 +1111,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c53e7e2..ff74c65 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -96,12 +96,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -220,12 +236,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -266,23 +282,6 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
DecodeAbort(ctx, buf, &parsed, xid);
break;
}
- case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index d714551..7553f84 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -145,7 +145,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_ABORT 0x20
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
-#define XLOG_XACT_ASSIGNMENT 0x50
+/* free opcode 0x50 */
/* free opcode 0x60 */
/* free opcode 0x70 */
@@ -188,15 +188,6 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XactCompletionForceSyncCommit(xinfo) \
((xinfo & XACT_COMPLETION_FORCE_SYNC_COMMIT) != 0)
-typedef struct xl_xact_assignment
-{
- TransactionId xtop; /* assigned XID's top-level XID */
- int nsubxacts; /* number of subtransaction XIDs */
- TransactionId xsub[FLEXIBLE_ARRAY_MEMBER]; /* assigned subxids */
-} xl_xact_assignment;
-
-#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
-
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
@@ -363,7 +354,6 @@ extern FullTransactionId GetTopFullTransactionId(void);
extern FullTransactionId GetTopFullTransactionIdIfAny(void);
extern FullTransactionId GetCurrentFullTransactionId(void);
extern FullTransactionId GetCurrentFullTransactionIdIfAny(void);
-extern void MarkCurrentTransactionIdLoggedIfAny(void);
extern bool SubTransactionIsActive(SubTransactionId subxid);
extern CommandId GetCurrentCommandId(bool used);
extern void SetParallelStartTimestamps(TimestampTz xact_ts, TimestampTz stmt_ts);
@@ -410,6 +400,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..060901d 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,8 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
+#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 60c76cb..d08c08a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3423,7 +3423,6 @@ xl_standby_locks
xl_tblspc_create_rec
xl_tblspc_drop_rec
xl_xact_abort
-xl_xact_assignment
xl_xact_commit
xl_xact_dbinfo
xl_xact_invals
--
1.8.3.1
0003-Issue-individual-invalidations-with-wal_level-logica.patch (application/octet-stream, 16.3K) download
From 2a2ac4be202b9226a12934ef7097764d6d8cb638 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:20:53 +0200
Subject: [PATCH 03/13] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 ++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 17 ++++++
src/backend/replication/logical/reorderbuffer.c | 52 +++++++++++++++-
src/backend/utils/cache/inval.c | 81 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 238 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 66fc8fb..9cff9f0 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -312,6 +317,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
xact_desc_abort(buf, XLogRecGetInfo(record), xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -336,7 +349,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ABORT_PREPARED:
id = "ABORT_PREPARED";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 33141fb..dc3633c 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5890,6 +5890,13 @@ xact_redo(XLogReaderState *record)
XLogRecGetOrigin(record));
LWLockRelease(TwoPhaseStateLock);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index ff74c65..c100054 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -282,6 +282,23 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
DecodeAbort(ctx, buf, &parsed, xid);
break;
}
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 6228140..08b4d4f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -460,6 +460,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1811,6 +1812,18 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2204,6 +2217,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2643,6 +2688,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2739,6 +2785,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3014,6 +3061,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
* although we don't check the memory limit when restoring the changes in
* this branch (we only do that when initially queueing the changes after
* decoding), because we will release the changes later, and that will
- * update the accounting too (subtracting the size from the counters).
- * And we don't want to underflow there.
+ * update the accounting too (subtracting the size from the counters). And
+ * we don't want to underflow there.
*/
ReorderBufferChangeMemoryUpdate(rb, change, true);
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..f921fdf 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
{
AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.sn.id = SHAREDINVALSNAPSHOT_ID;
+ msg.sn.dbId = dbId;
+ msg.sn.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -1501,3 +1558,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7553f84..b26d399 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -145,7 +145,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_ABORT 0x20
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
-/* free opcode 0x50 */
+#define XLOG_XACT_INVALIDATIONS 0x50
/* free opcode 0x60 */
/* free opcode 0x70 */
@@ -189,6 +189,22 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
((xinfo & XACT_COMPLETION_FORCE_SYNC_COMMIT) != 0)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4dcef80..82dcb7f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -437,6 +448,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch (application/octet-stream, 7.9K) download
From bab2c69894e1a22bfe9a96f452644065150bd4a9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 34 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++----------
2 files changed, 37 insertions(+), 30 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 08b4d4f..ca4b904 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -728,7 +728,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -748,7 +748,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -771,7 +771,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -831,7 +831,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -847,7 +847,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1057,7 +1057,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1086,7 +1086,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1252,7 +1252,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1300,7 +1300,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1315,7 +1315,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1332,7 +1332,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1972,7 +1972,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2120,7 +2120,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2298,7 +2298,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2315,7 +2315,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2335,7 +2335,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2520,7 +2520,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e6d17fb..afe690a 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 0db9e31ccc2537fc3c37edea1280210644905dc1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index da265f5..3230e45 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -69,6 +69,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -193,6 +208,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -867,6 +915,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 82dcb7f..e6d17fb 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
0006-Gracefully-handle-concurrent-aborts-of-uncommitted.patch (application/octet-stream, 13.7K) download
From 9777828b8f52fa1175fc1bf4ca2341b0bdb2cc9c Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:00:49 +0530
Subject: [PATCH 06/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 32 +++++++++++++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 142 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index e954482..6ce7878 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index ca4b904..1955bc5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -679,7 +679,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1496,6 +1496,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
ReorderBufferIterTXNState *volatile iterstate = NULL;
+ MemoryContext ccxt = CurrentMemoryContext;
txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
false);
@@ -1529,7 +1530,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1780,7 +1781,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1800,7 +1801,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Every time the CommandId is incremented, we could
@@ -1879,6 +1880,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
+ /*
+ * if the catalog scan access returned an error of
+ * rollback, then abort on the other side as well
+ */
+ if (errdata->sqlerrcode == ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ elog(LOG, "stopping decoding of %s (%u)",
+ txn->gid[0] != '\0'? txn->gid:"", txn->xid);
+ rb->abort(rb, txn, commit_lsn);
+ }
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1902,7 +1917,14 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* remove potential on-disk data, and deallocate */
ReorderBufferCleanupTXN(rb, txn);
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ FlushErrorState();
}
PG_END_TRY();
}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.2K) download
From 63de67f99e363f506502b490965e662e6374d56e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:02:37 +0530
Subject: [PATCH 07/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1074 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1111 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 537e681..76a105a 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1955bc5..c2d4677 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -358,6 +404,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -755,6 +804,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -851,6 +927,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -974,7 +1053,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1002,6 +1081,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1016,6 +1098,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1231,6 +1316,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1323,33 +1612,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1399,6 +1759,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1472,6 +1842,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1512,6 +1895,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1546,6 +1945,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1562,6 +1962,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -2037,6 +2447,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2172,8 +2589,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2181,6 +2607,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2192,19 +2619,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2232,6 +2668,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2307,6 +2744,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2321,6 +2761,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2425,6 +2882,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2444,15 +2933,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2737,6 +3257,504 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+
+ /*
+ * Every time the CommandId is incremented, we could
+ * see new catalog contents, so execute all
+ * invalidations.
+ */
+ ReorderBufferExecuteInvalidations(rb, txn);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index afe690a..f39b34f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.6K) download
From 71e1f6727e3bedcd372d3995022c53d9eb011b23 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:56:18 +0200
Subject: [PATCH 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 7e3ba8e..a1c2a9f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d85e831..5b3b095 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -67,7 +67,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -100,6 +101,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -194,6 +197,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -340,6 +363,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -356,7 +381,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -438,7 +464,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -705,11 +739,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -741,6 +778,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -753,7 +797,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -791,7 +835,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -828,7 +873,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 011076c..b22a053 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4106,6 +4106,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 65b3266..9baf61f 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 186057b..02574a0 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 3230e45..d5b5fe1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1153,7 +1153,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1198,7 +1198,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f737afb..3493b02 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1097,6 +1566,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1546,6 +2018,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1726,6 +2756,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 317c5d4..0faa701 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -48,16 +48,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -67,6 +93,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -87,16 +114,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -165,6 +202,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -177,6 +231,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -200,7 +255,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -220,6 +276,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -287,9 +364,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -315,19 +425,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -336,6 +453,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -364,14 +485,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -381,7 +502,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -390,7 +511,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -416,6 +537,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -436,13 +561,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -516,6 +642,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -626,6 +837,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 42da631..507661c 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index eb4a98c..1c0c020 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -944,6 +944,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..4d01f7e
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..1a8b8ff
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..04af090
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..6fecfe6
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..50990c1
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
0010-Track-statistics-for-streaming-spilling.patch (application/octet-stream, 13.9K) download
From 5404d4ad25d52190c7297642014e695277aee0a1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:01:30 +0200
Subject: [PATCH 10/13] Track statistics for streaming/spilling
---
doc/src/sgml/monitoring.sgml | 47 +++++++++++++++++++
src/backend/catalog/system_views.sql | 8 +++-
src/backend/replication/logical/reorderbuffer.c | 21 +++++++++
src/backend/replication/walsender.c | 62 ++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 6 +--
src/include/replication/reorderbuffer.h | 16 +++++++
src/include/replication/walsender_private.h | 10 ++++
src/test/regress/expected/rules.out | 10 +++-
8 files changed, 172 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 828e908..8de4fbd 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2121,6 +2121,53 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
with security-sensitive fields obfuscated.
</entry>
</row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a47..f8d0c4a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,13 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 963cbf3..5e91522 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -354,6 +354,14 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3001,6 +3009,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -3059,6 +3068,11 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillTxns += (rbtxn_is_serialized(txn)) ? 1 : 0;
+ rb->spillBytes += size;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
@@ -3763,6 +3777,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 1 : 0;
+ rb->streamBytes += txn->size;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1c0c020..14abd85 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1267,7 +1268,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1286,6 +1288,12 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk or
+ * streamed to subscriber (before being committed).
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2325,6 +2333,12 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3236,7 +3250,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3291,6 +3305,12 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3311,6 +3331,12 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3392,6 +3418,16 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3628,3 +3664,25 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..9a508bf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f39b34f..53d9440 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -509,6 +509,22 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions streamed or spilled to disk.
+ *
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..f726f25 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,16 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 210e9cd..836abf0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1951,9 +1951,15 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0009-Extend-the-concurrent-abort-handling-for-in-progress.patch (application/octet-stream, 2.1K) download
From 2a0712fb830a6be20296d18fb821862e302aeb8b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:04:09 +0530
Subject: [PATCH 09/13] Extend the concurrent abort handling for in-progress
transaction
---
src/backend/replication/logical/reorderbuffer.c | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c2d4677..963cbf3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3280,6 +3280,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3410,6 +3411,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3722,6 +3730,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3740,7 +3751,14 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ FlushErrorState();
}
PG_END_TRY();
--
1.8.3.1
0011-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 16.8K) download
From 0bc62e1df19e19198d20330f54471abfe9618abd Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:16 +0200
Subject: [PATCH 11/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 4 ++--
src/test/subscription/t/010_stream_subxact.pl | 4 ++--
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 4 ++--
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 4 ++--
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 6 +++---
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 3ad00ea..78263c7 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 4d01f7e..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index 1a8b8ff..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 04af090..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 6fecfe6..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index 50990c1..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -1,4 +1,4 @@
-# Test behavior with streaming transaction exceeding logical_work_mem
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
use strict;
use warnings;
use PostgresNode;
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
0012-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1010B) download
From 2fbbd9211397c5c79a25aa6b8160234d6fe4dfe6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 12/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5e91522..29da34d 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
0013-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From c3146318b04d5a6beb139c5c8120cdeee0a8a8b6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 13/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v3-0014-BGWorkers-pool-for-streamed-transactions-apply.patch (application/octet-stream, 60.2K) download
From 9623e79695eb018f1e359e1a2b4b1be22a22d6a3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 11:34:35 +0530
Subject: [PATCH] BGWorkers pool for streamed transactions apply
---
src/backend/postmaster/bgworker.c | 3 +
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/logical/proto.c | 17 +-
src/backend/replication/logical/worker.c | 1782 ++++++++++++++++--------------
src/include/pgstat.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/include/replication/logicalworker.h | 1 +
7 files changed, 936 insertions(+), 877 deletions(-)
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index b66b517..2e18083 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "LogicalApplyBgwMain", LogicalApplyBgwMain
}
};
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index b22a053..46a75f2 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3810,6 +3810,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING:
event_name = "Hash/GrowBuckets/Reinserting";
break;
+ case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY:
+ event_name = "LogicalApplyWorkerReady";
+ break;
case WAIT_EVENT_LOGICAL_SYNC_DATA:
event_name = "LogicalSyncData";
break;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 5a379fb..d615b1f 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -788,14 +788,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
pq_sendint64(out, txn->commit_time);
}
-TransactionId
+void
logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
{
- TransactionId xid;
uint8 flags;
- xid = pq_getmsgint(in, 4);
-
/* read flags (unused for now) */
flags = pq_getmsgbyte(in);
@@ -806,8 +803,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
commit_data->commit_lsn = pq_getmsgint64(in);
commit_data->end_lsn = pq_getmsgint64(in);
commit_data->committime = pq_getmsgint64(in);
-
- return xid;
}
void
@@ -822,13 +817,3 @@ logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
pq_sendint32(out, xid);
pq_sendint32(out, subxid);
}
-
-void
-logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid)
-{
- Assert(xid && subxid);
-
- *xid = pq_getmsgint(in, 4);
- *subxid = pq_getmsgint(in, 4);
-}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3493b02..7458ac0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -81,11 +81,15 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/shm_mq.h"
+#include "storage/shm_toc.h"
+#include "storage/spin.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
@@ -101,6 +105,54 @@
#include "utils/timeout.h"
#define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */
+#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change
+
+typedef struct ParallelState
+{
+ slock_t mutex;
+ // ConditionVariable cv;
+ bool attached;
+ bool ready;
+ bool finished;
+ Oid database_id;
+ Oid authenticated_user_id;
+ Oid subid;
+ Oid stream_xid;
+ uint32 n;
+} ParallelState;
+
+typedef struct WorkerState
+{
+ TransactionId xid;
+ BackgroundWorkerHandle *handle;
+ shm_mq_handle *mq_handle;
+ dsm_segment *dsm_seg;
+ ParallelState volatile *pstate;
+} WorkerState;
+
+/* Apply workers hash table (initialized on first use) */
+static HTAB *ApplyWorkersHash = NULL;
+static WorkerState **ApplyWorkersIdleList = NULL;
+static uint32 pool_size = 10; /* MaxConnections default? */
+static uint32 nworkers = 0;
+static uint32 nfreeworkers = 0;
+
+/* Fields valid only for apply background workers */
+bool isLogicalApplyWorker = false;
+volatile ParallelState *MyParallelState = NULL;
+
+/* Worker setup and interactions */
+static void setup_dsm(WorkerState *wstate);
+static void setup_background_worker(WorkerState *wstate);
+static void cleanup_background_worker(dsm_segment *seg, Datum arg);
+static void handle_sigterm(SIGNAL_ARGS);
+
+static bool check_worker_status(WorkerState *wstate);
+static void wait_for_worker(WorkerState *wstate);
+static void wait_for_worker_to_finish(WorkerState *wstate);
+
+static WorkerState * find_or_start_worker(TransactionId xid, bool start);
+static void stop_worker(WorkerState *wstate);
typedef struct FlushPosition
{
@@ -129,47 +181,13 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
-/* fields valid only when processing streamed transaction */
+/* Fields valid only when processing streamed transaction */
bool in_streamed_transaction = false;
static TransactionId stream_xid = InvalidTransactionId;
-static int stream_fd = -1;
-
-typedef struct SubXactInfo
-{
- TransactionId xid; /* XID of the subxact */
- off_t offset; /* offset in the file */
-} SubXactInfo;
-
-static uint32 nsubxacts = 0;
-static uint32 nsubxacts_max = 0;
-static SubXactInfo * subxacts = NULL;
-static TransactionId subxact_last = InvalidTransactionId;
-
-static void subxact_filename(char *path, Oid subid, TransactionId xid);
-static void changes_filename(char *path, Oid subid, TransactionId xid);
-
-/*
- * Information about subtransactions of a given toplevel transaction.
- */
-static void subxact_info_write(Oid subid, TransactionId xid);
-static void subxact_info_read(Oid subid, TransactionId xid);
-static void subxact_info_add(TransactionId xid);
-
-/*
- * Serialize and deserialize changes for a toplevel transaction.
- */
-static void stream_cleanup_files(Oid subid, TransactionId xid);
-static void stream_open_file(Oid subid, TransactionId xid, bool first);
-static void stream_write_change(char action, StringInfo s);
-static void stream_close_file(void);
-
-/*
- * Array of serialized XIDs.
- */
-static int nxids = 0;
-static int maxnxids = 0;
-static TransactionId *xids = NULL;
+static TransactionId current_xid = InvalidTransactionId;
+static TransactionId prev_xid = InvalidTransactionId;
+static uint32 nchanges = 0;
static bool handle_streamed_transaction(const char action, StringInfo s);
@@ -185,6 +203,16 @@ static volatile sig_atomic_t got_SIGHUP = false;
/* prototype needed because of stream_commit */
static void apply_dispatch(StringInfo s);
+// /* Debug only */
+// static void
+// iter_sleep(int seconds)
+// {
+// for (int i = 0; i < seconds; i++)
+// {
+// pg_usleep(1 * 1000L * 1000L);
+// }
+// }
+
/*
* Should this worker apply changes for given relation.
*
@@ -237,6 +265,107 @@ ensure_transaction(void)
}
/*
+ * Look up worker inside ApplyWorkersHash for requested xid.
+ * Throw error if not found or start a new one if start=true is passed.
+ */
+static WorkerState *
+find_or_start_worker(TransactionId xid, bool start)
+{
+ bool found;
+ WorkerState *entry = NULL;
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* First time through, initialize apply workers hashtable */
+ if (ApplyWorkersHash == NULL)
+ {
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(WorkerState);
+ ctl.hcxt = ApplyContext; /* Allocate ApplyWorkersHash in the ApplyContext */
+ ApplyWorkersHash = hash_create("logical apply workers hash", 8,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ }
+
+ Assert(ApplyWorkersHash != NULL);
+
+ /*
+ * Find entry for requested transaction.
+ */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found);
+
+ if (!found && start)
+ {
+ /* If there is at least one worker in the idle list, then take one. */
+ if (nfreeworkers > 0)
+ {
+ char action = 'R';
+
+ Assert(ApplyWorkersIdleList != NULL);
+
+ entry = ApplyWorkersIdleList[nfreeworkers - 1];
+ if (!hash_update_hash_key(ApplyWorkersHash,
+ (void *) entry,
+ (void *) &xid))
+ elog(ERROR, "could not reassign apply worker #%u entry from xid %u to xid %u",
+ entry->pstate->n, entry->xid, xid);
+
+ entry->xid = xid;
+ entry->pstate->finished = false;
+ entry->pstate->stream_xid = xid;
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+
+ ApplyWorkersIdleList[--nfreeworkers] = NULL;
+ }
+ else
+ {
+ /* No entry in hash and no idle workers. Create a new one. */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found);
+ entry->xid = xid;
+ setup_background_worker(entry);
+
+ if (nworkers == pool_size)
+ {
+ ApplyWorkersIdleList = repalloc(ApplyWorkersIdleList, pool_size + 10);
+ pool_size += 10;
+ }
+ }
+ }
+ else if (!found && !start)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not find logical apply worker for xid %u", xid)));
+ else
+ elog(DEBUG5, "there is an existing logical apply worker for xid %u", xid);
+
+ Assert(entry != NULL);
+
+ return entry;
+}
+
+/*
+ * Gracefully teardown apply worker.
+ */
+static void
+stop_worker(WorkerState *wstate)
+{
+ /*
+ * Sending zero-length data to worker in order to stop it.
+ */
+ shm_mq_send(wstate->mq_handle, 0, NULL, false);
+
+ elog(LOG, "detaching DSM of apply worker #%u for xid %u",
+ wstate->pstate->n, wstate->xid);
+ dsm_detach(wstate->dsm_seg);
+
+ /* Delete worker entry */
+ (void) hash_search(ApplyWorkersHash, &wstate->xid, HASH_REMOVE, NULL);
+}
+
+/*
* Handle streamed transactions.
*
* If in streaming mode (receiving a block of streamed transaction), we
@@ -248,12 +377,12 @@ static bool
handle_streamed_transaction(const char action, StringInfo s)
{
TransactionId xid;
+ WorkerState *entry;
/* not in streaming mode */
- if (!in_streamed_transaction)
+ if (!in_streamed_transaction || isLogicalApplyWorker)
return false;
- Assert(stream_fd != -1);
Assert(TransactionIdIsValid(stream_xid));
/*
@@ -264,11 +393,16 @@ handle_streamed_transaction(const char action, StringInfo s)
Assert(TransactionIdIsValid(xid));
- /* Add the new subxact to the array (unless already there). */
- subxact_info_add(xid);
+ /*
+ * Find worker for requested xid.
+ */
+ entry = find_or_start_worker(stream_xid, false);
- /* write the change to the current file */
- stream_write_change(action, s);
+ // elog(LOG, "sending message of length=%d and raw=%s, action=%s", s->len, s->data, (char *) &action);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
+ nchanges += 1;
+
+ // iter_sleep(3600);
return true;
}
@@ -624,7 +758,8 @@ apply_handle_origin(StringInfo s)
static void
apply_handle_stream_start(StringInfo s)
{
- bool first_segment;
+ bool first_segment;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
@@ -633,17 +768,16 @@ apply_handle_stream_start(StringInfo s)
/* extract XID of the top-level transaction */
stream_xid = logicalrep_read_stream_start(s, &first_segment);
+ nchanges = 0;
- /* open the spool file for this transaction */
- stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, true);
- /*
- * if this is not the first segment, open existing file
- *
- * XXX Note that the cleanup is performed by stream_open_file.
- */
- if (!first_segment)
- subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+ SpinLockAcquire(&entry->pstate->mutex);
+ entry->pstate->ready = false;
+ SpinLockRelease(&entry->pstate->mutex);
+
+ elog(LOG, "starting streaming of xid %u", stream_xid);
pgstat_report_activity(STATE_RUNNING, NULL);
}
@@ -654,16 +788,19 @@ apply_handle_stream_start(StringInfo s)
static void
apply_handle_stream_stop(StringInfo s)
{
+ WorkerState *entry;
+ char action = 'E';
+
Assert(in_streamed_transaction);
- /*
- * Close the file with serialized changes, and serialize information about
- * subxacts for the toplevel transaction.
- */
- subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
- stream_close_file();
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
+
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ wait_for_worker(entry);
in_streamed_transaction = false;
+ elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges);
pgstat_report_activity(STATE_IDLE, NULL);
}
@@ -676,96 +813,67 @@ apply_handle_stream_abort(StringInfo s)
{
TransactionId xid;
TransactionId subxid;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
- logicalrep_read_stream_abort(s, &xid, &subxid);
-
- /*
- * If the two XIDs are the same, it's in fact abort of toplevel xact, so
- * just delete the files with serialized info.
- */
- if (xid == subxid)
+ if(isLogicalApplyWorker)
{
- char path[MAXPGPATH];
+ subxid = pq_getmsgint(s, 4);
- /*
- * XXX Maybe this should be an error instead? Can we receive abort for
- * a toplevel transaction we haven't received?
- */
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u",
+ MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId())));
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ if (subxid == stream_xid)
+ AbortCurrentTransaction();
+ else
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", subxid);
- if (unlink(path) < 0)
- ereport(ERROR,
+ ereport(LOG,
(errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname)));
- subxact_filename(path, MyLogicalRepWorker->subid, xid);
-
- if (unlink(path) < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ RollbackToSavepoint(spname);
+ CommitTransactionCommand();
+ // RollbackAndReleaseCurrentSubTransaction();
- return;
+ pfree(spname);
+ }
}
else
{
- /*
- * OK, so it's a subxact. We need to read the subxact file for the
- * toplevel transaction, determine the offset tracked for the subxact,
- * and truncate the file with changes. We also remove the subxacts
- * with higher offsets (or rather higher XIDs).
- *
- * We intentionally scan the array from the tail, because we're likely
- * aborting a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- *
- * XXX Or perhaps we can rely on the aborts to arrive in the reverse
- * order, i.e. from the inner-most subxact (when nested)? In which
- * case we could simply check the last element.
- */
+ xid = pq_getmsgint(s, 4);
+ subxid = pq_getmsgint(s, 4);
- int64 i;
- int64 subidx;
- bool found = false;
- char path[MAXPGPATH];
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
- subidx = -1;
- subxact_info_read(MyLogicalRepWorker->subid, xid);
+ elog(LOG, "processing abort request of streamed transaction xid %u, subxid %u",
+ xid, subxid);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- /* FIXME optimize the search by bsearch on sorted data */
- for (i = nsubxacts; i > 0; i--)
+ if (subxid == stream_xid)
{
- if (subxacts[i - 1].xid == subxid)
- {
- subidx = (i - 1);
- found = true;
- break;
- }
- }
-
- /* We should not receive aborts for unknown subtransactions. */
- Assert(found);
+ char action = 'F';
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ // shm_mq_send(entry->mq_handle, 0, NULL, false);
- /* OK, truncate the file at the right offset. */
- Assert((subidx >= 0) && (subidx < nsubxacts));
+ wait_for_worker_to_finish(entry);
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
- if (truncate(path, subxacts[subidx].offset))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not truncate file \"%s\": %m", path)));
+ // elog(LOG, "detaching DSM of apply worker for xid=%u\n", entry->xid);
+ // dsm_detach(entry->dsm_seg);
- /* discard the subxacts added later */
- nsubxacts = subidx;
-
- /* write the updated subxact list */
- subxact_info_write(MyLogicalRepWorker->subid, xid);
+ // /* Delete worker entry */
+ // (void) hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL);
+ }
}
}
@@ -775,159 +883,56 @@ apply_handle_stream_abort(StringInfo s)
static void
apply_handle_stream_commit(StringInfo s)
{
- int fd;
TransactionId xid;
- StringInfoData s2;
- int nchanges;
-
- char path[MAXPGPATH];
- char *buffer = NULL;
+ WorkerState *entry;
LogicalRepCommitData commit_data;
- MemoryContext oldcxt;
-
- Assert(!in_streamed_transaction);
-
- xid = logicalrep_read_stream_commit(s, &commit_data);
-
- elog(DEBUG1, "received commit for streamed transaction %u", xid);
-
- /* open the spool file for the committed transaction */
- changes_filename(path, MyLogicalRepWorker->subid, xid);
-
- elog(DEBUG1, "replaying changes from file '%s'", path);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
+ if (isLogicalApplyWorker)
{
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- }
-
- /* XXX Should this be allocated in another memory context? */
+ // logicalrep_read_stream_commit(s, &commit_data);
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- buffer = palloc(8192);
- initStringInfo(&s2);
-
- MemoryContextSwitchTo(oldcxt);
-
- ensure_transaction();
-
- /*
- * Make sure the handle apply_dispatch methods are aware we're in a remote
- * transaction.
- */
- in_remote_transaction = true;
- pgstat_report_activity(STATE_RUNNING, NULL);
-
- /*
- * Read the entries one by one and pass them through the same logic as in
- * apply_dispatch.
- */
- nchanges = 0;
- while (true)
+ CommitTransactionCommand();
+ }
+ else
{
- int nbytes;
- int len;
-
- /* read length of the on-disk record */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- nbytes = read(fd, &len, sizeof(len));
- pgstat_report_wait_end();
-
- /* have we reached end of the file? */
- if (nbytes == 0)
- break;
-
- /* do we have a correct length? */
- if (nbytes != sizeof(len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
-
- Assert(len > 0);
+ char action = 'F';
- /* make sure we have sufficiently large buffer */
- buffer = repalloc(buffer, len);
-
- /* and finally read the data into the buffer */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- if (read(fd, buffer, len) != len)
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
- pgstat_report_wait_end();
+ Assert(!in_streamed_transaction);
- /* copy the buffer to the stringinfo and call apply_dispatch */
- resetStringInfo(&s2);
- appendBinaryStringInfo(&s2, buffer, len);
+ xid = pq_getmsgint(s, 4);
+ logicalrep_read_stream_commit(s, &commit_data);
- /* Ensure we are reading the data into our memory context. */
- oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
- apply_dispatch(&s2);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(xid, false);
- MemoryContextReset(ApplyMessageContext);
+ /* Send commit message */
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- MemoryContextSwitchTo(oldcxt);
+ /* Notify worker, that we are done with this xact */
+ shm_mq_send(entry->mq_handle, 1, &action, false);
- nchanges++;
+ wait_for_worker_to_finish(entry);
- if (nchanges % 1000 == 0)
- elog(DEBUG1, "replayed %d changes from file '%s'",
- nchanges, path);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
/*
- * send feedback to upstream
- *
- * XXX Probably should send a valid LSN. But which one?
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
*/
- send_feedback(InvalidXLogRecPtr, false, false);
- }
-
- CloseTransientFile(fd);
-
- /*
- * Update origin state so we can restart streaming from correct
- * position in case of crash.
- */
- replorigin_session_origin_lsn = commit_data.end_lsn;
- replorigin_session_origin_timestamp = commit_data.committime;
-
- CommitTransactionCommand();
- pgstat_report_stat(false);
-
- store_flush_position(commit_data.end_lsn);
-
- elog(DEBUG1, "replayed %d (all) changes from file '%s'",
- nchanges, path);
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
- in_remote_transaction = false;
- pgstat_report_activity(STATE_IDLE, NULL);
+ pgstat_report_stat(false);
- /* unlink the files with serialized changes and subxact info */
- stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ store_flush_position(commit_data.end_lsn);
- pfree(buffer);
- pfree(s2.data);
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
}
/*
@@ -946,6 +951,8 @@ apply_handle_relation(StringInfo s)
if (handle_streamed_transaction('R', s))
return;
+ // iter_sleep(3600);
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -1386,6 +1393,38 @@ apply_dispatch(StringInfo s)
{
char action = pq_getmsgbyte(s);
+ if (isLogicalApplyWorker)
+ {
+ /*
+ * Inside logical apply worker we can figure out that new subtransaction
+ * was started if new change arrived with different xid. In that case we
+ * can define named savepoint, so that we were able to commit/rollback it
+ * separately later.
+ *
+ * Special case is if the first change comes from subtransuction, then
+ * we check that current_xid differs from stream_xid.
+ */
+ current_xid = pq_getmsgint(s, 4);
+
+ if (current_xid != stream_xid
+ && ((TransactionIdIsValid(prev_xid) && current_xid != prev_xid)
+ || !TransactionIdIsValid(prev_xid)))
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", current_xid);
+
+ elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname);
+
+ DefineSavepoint(spname);
+ CommitTransactionCommand();
+ // BeginInternalSubTransaction(NULL);
+ }
+
+ prev_xid = current_xid;
+ }
+ // else
+ // elog(LOG, "Logical worker: applying dispatch for action=%s", (char *) &action);
+
switch (action)
{
/* BEGIN */
@@ -1414,6 +1453,7 @@ apply_dispatch(StringInfo s)
break;
/* RELATION */
case 'R':
+ // elog(LOG, "%s worker: applying dispatch for action=R", isLogicalApplyWorker ? "Apply" : "Logical");
apply_handle_relation(s);
break;
/* TYPE */
@@ -1544,12 +1584,18 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
static void
worker_onexit(int code, Datum arg)
{
- int i;
+ HASH_SEQ_STATUS status;
+ WorkerState *entry;
- elog(LOG, "cleanup files for %d transactions", nxids);
-
- for (i = nxids-1; i >= 0; i--)
- stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+ if (ApplyWorkersHash != NULL)
+ {
+ hash_seq_init(&status, ApplyWorkersHash);
+ while ((entry = (WorkerState *) hash_seq_search(&status)) != NULL)
+ {
+ stop_worker(entry);
+ }
+ hash_seq_term(&status);
+ }
}
/*
@@ -1572,6 +1618,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
+ ApplyWorkersIdleList = palloc(sizeof(WorkerState *) * pool_size);
+
for (;;)
{
pgsocket fd = PGINVALID_SOCKET;
@@ -1883,8 +1931,9 @@ maybe_reread_subscription(void)
Subscription *newsub;
bool started_tx = false;
+ // TODO Probably we have to handle subscription reread in apply workers too.
/* When cache state is valid there is nothing to do here. */
- if (MySubscriptionValid)
+ if (MySubscriptionValid || isLogicalApplyWorker)
return;
/* This function might be called inside or outside of transaction. */
@@ -2018,608 +2067,50 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
-/*
- * subxact_info_write
- * Store information about subxacts for a toplevel transaction.
- *
- * For each subxact we store offset of it's first change in the main file.
- * The file is always over-written as a whole, and we also include CRC32C
- * checksum of the information.
- *
- * XXX We should only store subxacts that were not aborted yet.
- *
- * XXX Maybe we should only include the checksum when the cluster is
- * initialized with checksums?
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- */
+/* SIGHUP: set flag to reload configuration at next convenient time */
static void
-subxact_info_write(Oid subid, TransactionId xid)
+logicalrep_worker_sighup(SIGNAL_ARGS)
{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- Size len;
-
- Assert(TransactionIdIsValid(xid));
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create file \"%s\": %m",
- path)));
- return;
- }
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* compute the checksum */
- INIT_CRC32C(checksum);
- COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum, (char *) subxacts, len);
- FIN_CRC32C(checksum);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
-
- if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
+ int save_errno = errno;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ got_SIGHUP = true;
- if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ errno = save_errno;
+}
- if ((len > 0) && (write(fd, subxacts, len) != len))
- {
- int save_errno = errno;
+/* Logical Replication Apply worker entry point */
+void
+ApplyWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ MemoryContext oldctx;
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos;
+ char *myslotname;
+ WalRcvStreamOptions options;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
- pgstat_report_wait_end();
+ /* Setup signal handling */
+ pqsignal(SIGHUP, logicalrep_worker_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
/*
- * We don't need to fsync or anything, as we'll recreate the files after a
- * crash from scratch. So just close the file.
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
*/
- CloseTransientFile(fd);
- /*
- * But we free the memory allocated for subxact info. There might be one
- * exceptional transaction with many subxacts, and we don't want to keep
- * the memory allocated forewer.
- *
- */
- if (subxacts)
- pfree(subxacts);
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
- subxacts = NULL;
- subxact_last = InvalidTransactionId;
- nsubxacts = 0;
- nsubxacts_max = 0;
-}
-
-/*
- * subxact_info_read
- * Restore information about subxacts of a streamed transaction.
- *
- * Read information about subxacts into the global variables, and while
- * reading the information verify the checksum.
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_read(Oid subid, TransactionId xid)
-{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- uint32 checksum_new;
- Size len;
- MemoryContext oldctx;
-
- Assert(TransactionIdIsValid(xid));
- Assert(!subxacts);
- Assert(nsubxacts == 0);
- Assert(nsubxacts_max == 0);
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- /* read the checksum */
- if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- /* read number of subxact items */
- if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* we keep the maximum as a power of 2 */
- nsubxacts_max = 1 << my_log2(nsubxacts);
-
- /* subxacts are long-lived */
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- if ((len > 0) && ((read(fd, subxacts, len)) != len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- /* recompute the checksum */
- INIT_CRC32C(checksum_new);
- COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum_new, (char *) subxacts, len);
- FIN_CRC32C(checksum_new);
-
- if (checksum_new != checksum)
- ereport(ERROR,
- (errmsg("checksum failure when reading subxacts")));
-
- CloseTransientFile(fd);
-}
-
-/*
- * subxact_info_add
- * Add information about a subxact (offset in the main file).
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_add(TransactionId xid)
-{
- int64 i;
-
- /*
- * If the XID matches the toplevel transaction, we don't want to add it.
- */
- if (stream_xid == xid)
- return;
-
- /*
- * In most cases we're checking the same subxact as we've already seen in
- * the last call, so make ure just ignore it (this change comes later).
- */
- if (subxact_last == xid)
- return;
-
- /* OK, remember we're processing this XID. */
- subxact_last = xid;
-
- /*
- * Check if the transaction is already present in the array of subxact. We
- * intentionally scan the array from the tail, because we're likely adding
- * a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- */
- for (i = nsubxacts; i > 0; i--)
- {
- /* found, so we're done */
- if (subxacts[i - 1].xid == xid)
- return;
- }
-
- /* This is a new subxact, so we need to add it to the array. */
-
- if (nsubxacts == 0)
- {
- MemoryContext oldctx;
-
- nsubxacts_max = 128;
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
- }
- else if (nsubxacts == nsubxacts_max)
- {
- nsubxacts_max *= 2;
- subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
- }
-
- subxacts[nsubxacts].xid = xid;
- subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
-
- nsubxacts++;
-}
-
-/* format filename for file containing the info about subxacts */
-static void
-subxact_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
- tempdirpath, subid, xid);
-}
-
-/* format filename for file containing serialized changes */
-static void
-changes_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
- tempdirpath, subid, xid);
-}
-
-/*
- * stream_cleanup_files
- * Cleanup files for a subscription / toplevel transaction.
- *
- * Remove files with serialized changes and subxact info for a particular
- * toplevel transaction. Each subscription has a separate set of files.
- *
- * Note: The files may not exists, so handle ENOENT as non-error.
- *
- * TODO: Add missing_ok flag to specify in which cases it's OK not to
- * find the files, and when it's an error.
- */
-static void
-stream_cleanup_files(Oid subid, TransactionId xid)
-{
- int i;
- char path[MAXPGPATH];
- bool found = false;
-
- subxact_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- changes_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- /*
- * Cleanup the XID from the array - find the XID in the array and
- * remove it by shifting all the remaining elements. The array is
- * bound to be fairly small (maximum number of in-progress xacts,
- * so max_connections + max_prepared_transactions) so simply loop
- * through the array and find index of the XID. Then move the rest
- * of the array by one element to the left.
- *
- * Notice we also call this from stream_open_file for first segment
- * of each transaction, to deal with possible left-overs after a
- * crash, so it's entirely possible not to find the XID in the
- * array here. In that case we don't remove anything.
- *
- * XXX Perhaps it'd be better to handle this automatically after a
- * restart, instead of doing it over and over for each transaction.
- */
- for (i = 0; i < nxids; i++)
- {
- if (xids[i] == xid)
- {
- found = true;
- break;
- }
- }
-
- if (!found)
- return;
-
- /*
- * Move the last entry from the array to the place. We don't keep
- * the streamed transactions sorted or anything - we only expect
- * a few of them in progress (max_connections + max_prepared_xacts)
- * so linear search is just fine.
- */
- xids[i] = xids[nxids-1];
- nxids--;
-}
-
-/*
- * stream_open_file
- * Open file we'll use to serialize changes for a toplevel transaction.
- *
- * Open a file for streamed changes from a toplevel transaction identified
- * by stream_xid (global variable). If it's the first chunk of streamed
- * changes for this transaction, perform cleanup by removing existing
- * files after a possible previous crash.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_open_file(Oid subid, TransactionId xid, bool first_segment)
-{
- char path[MAXPGPATH];
- int flags;
-
- Assert(in_streamed_transaction);
- Assert(OidIsValid(subid));
- Assert(TransactionIdIsValid(xid));
- Assert(stream_fd == -1);
-
- /*
- * If this is the first segment for this transaction, try removing
- * existing files (if there are any, possibly after a crash).
- */
- if (first_segment)
- {
- MemoryContext oldcxt;
-
- /* XXX make sure there are no previous files for this transaction */
- stream_cleanup_files(subid, xid);
-
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- /*
- * We need to remember the XIDs we spilled to files, so that we can
- * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
- *
- * The number of XIDs we may need to track is fairly small, because
- * we can only stream toplevel xacts (so limited by max_connections
- * and max_prepared_transactions), and we only stream the large ones.
- * So we simply keep the XIDs in an unsorted array. If the number of
- * xacts gets large for some reason (e.g. very high max_connections),
- * a more elaborate approach might be better - e.g. sorted array, to
- * speed-up the lookups.
- */
- if (nxids == maxnxids) /* array of XIDs is full */
- {
- if (!xids)
- {
- maxnxids = 64;
- xids = palloc(maxnxids * sizeof(TransactionId));
- }
- else
- {
- maxnxids = 2 * maxnxids;
- xids = repalloc(xids, maxnxids * sizeof(TransactionId));
- }
- }
-
- xids[nxids++] = xid;
-
- MemoryContextSwitchTo(oldcxt);
- }
-
- changes_filename(path, subid, xid);
-
- elog(DEBUG1, "opening file '%s' for streamed changes", path);
-
- /*
- * If this is the first streamed segment, the file must not exist, so
- * make sure we're the ones creating it. Otherwise just open the file
- * for writing, in append mode.
- */
- if (first_segment)
- flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
- else
- flags = (O_WRONLY | O_APPEND | PG_BINARY);
-
- stream_fd = OpenTransientFile(path, flags);
-
- if (stream_fd < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
-}
-
-/*
- * stream_close_file
- * Close the currently open file with streamed changes.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_close_file(void)
-{
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- CloseTransientFile(stream_fd);
-
- stream_xid = InvalidTransactionId;
- stream_fd = -1;
-}
-
-/*
- * stream_write_change
- * Serialize a change to a file for the current toplevel transaction.
- *
- * The change is serialied in a simple format, with length (not including
- * the length), action code (identifying the message type) and message
- * contents (without the subxact TransactionId value).
- *
- * XXX The subxact file includes CRC32C of the contents. Maybe we should
- * include something like that here too, but doing so will not be as
- * straighforward, because we write the file in chunks.
- */
-static void
-stream_write_change(char action, StringInfo s)
-{
- int len;
-
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- /* total on-disk size, including the action type character */
- len = (s->len - s->cursor) + sizeof(char);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
-
- /* first write the size */
- if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* then the action */
- if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* and finally the remaining part of the buffer (after the XID) */
- len = (s->len - s->cursor);
-
- if (write(stream_fd, &s->data[s->cursor], len) != len)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- pgstat_report_wait_end();
-}
-
-/* SIGHUP: set flag to reload configuration at next convenient time */
-static void
-logicalrep_worker_sighup(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- got_SIGHUP = true;
-
- /* Waken anything waiting on the process latch */
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
-/* Logical Replication Apply worker entry point */
-void
-ApplyWorkerMain(Datum main_arg)
-{
- int worker_slot = DatumGetInt32(main_arg);
- MemoryContext oldctx;
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos;
- char *myslotname;
- WalRcvStreamOptions options;
-
- /* Attach to slot */
- logicalrep_worker_attach(worker_slot);
-
- /* Setup signal handling */
- pqsignal(SIGHUP, logicalrep_worker_sighup);
- pqsignal(SIGTERM, die);
- BackgroundWorkerUnblockSignals();
-
- /*
- * We don't currently need any ResourceOwner in a walreceiver process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
-
- /* Initialise stats to a sanish value */
- MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
- MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
-
- /* Load the libpq-specific functions */
- load_file("libpqwalreceiver", false);
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
/* Run as replica session replication role. */
SetConfigOption("session_replication_role", "replica",
@@ -2775,3 +2266,580 @@ IsLogicalWorker(void)
{
return MyLogicalRepWorker != NULL;
}
+
+/*
+ * Apply Background Worker main loop.
+ */
+void
+LogicalApplyBgwMain(Datum main_arg)
+{
+ volatile ParallelState *pst;
+
+ dsm_segment *seg;
+ shm_toc *toc;
+ PGPROC *registrant;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ shm_mq_result shmq_res;
+ // ConditionVariable cv;
+ LogicalRepWorker lrw;
+ MemoryContext oldcontext;
+
+ MemoryContextSwitchTo(TopMemoryContext);
+
+ /* Load the subscription into persistent memory context. */
+ ApplyContext = AllocSetContextCreate(TopMemoryContext,
+ "ApplyContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldcontext = MemoryContextSwitchTo(ApplyContext);
+
+ /*
+ * Init the ApplyMessageContext which we clean up after each replication
+ * protocol message.
+ */
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ isLogicalApplyWorker = true;
+
+ /*
+ * Establish signal handlers.
+ *
+ * We want CHECK_FOR_INTERRUPTS() to kill off this worker process just as
+ * it would a normal user backend. To make that happen, we establish a
+ * signal handler that is a stripped-down version of die().
+ */
+ pqsignal(SIGTERM, handle_sigterm);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Connect to the dynamic shared memory segment.
+ *
+ * The backend that registered this worker passed us the ID of a shared
+ * memory segment to which we must attach for further instructions. In
+ * order to attach to dynamic shared memory, we need a resource owner.
+ * Once we've mapped the segment in our address space, attach to the table
+ * of contents so we can locate the various data structures we'll need to
+ * find within the segment.
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker");
+ seg = dsm_attach(DatumGetInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("unable to map dynamic shared memory segment")));
+ toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg));
+ if (toc == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("bad magic number in dynamic shared memory segment")));
+
+ /*
+ * Acquire a worker number.
+ *
+ * By convention, the process registering this background worker should
+ * have stored the control structure at key 0. We look up that key to
+ * find it. Our worker number gives our identity: there may be just one
+ * worker involved in this parallel operation, or there may be many.
+ */
+ pst = shm_toc_lookup(toc, 0, false);
+ MyParallelState = pst;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->attached = true;
+ SpinLockRelease(&pst->mutex);
+
+ /*
+ * Attach to the message queue.
+ */
+ mq = shm_toc_lookup(toc, 1, false);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ /* Restore database connection. */
+ BackgroundWorkerInitializeConnectionByOid(pst->database_id,
+ pst->authenticated_user_id, 0);
+
+ /*
+ * Set the client encoding to the database encoding, since that is what
+ * the leader will expect.
+ */
+ SetClientEncoding(GetDatabaseEncoding());
+
+ lrw.subid = pst->subid;
+ MyLogicalRepWorker = &lrw;
+
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+ // PushActiveSnapshot(GetTransactionSnapshot());
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ /*
+ * Indicate that we're fully initialized and ready to begin the main part
+ * of the parallel operation.
+ *
+ * Once we signal that we're ready, the user backend is entitled to assume
+ * that our on_dsm_detach callbacks will fire before we disconnect from
+ * the shared memory segment and exit. Generally, that means we must have
+ * attached to all relevant dynamic shared memory data structures by now.
+ */
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ // cv = pst->cv;
+ // if (pst->workers_ready == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+ elog(LOG, "[Apply BGW #%u] started", pst->n);
+
+ registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ SetLatch(®istrant->procLatch);
+
+ for (;;)
+ {
+ void *data;
+ Size len;
+ StringInfoData s;
+ MemoryContext oldctx;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Ensure we are reading the data into our memory context. */
+ oldctx = MemoryContextSwitchTo(ApplyMessageContext);
+
+ shmq_res = shm_mq_receive(mqh, &len, &data, false);
+
+ if (shmq_res != SHM_MQ_SUCCESS)
+ break;
+
+ if (len == 0)
+ {
+ elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n);
+ break;
+ }
+ else
+ {
+ s.cursor = 0;
+ s.maxlen = -1;
+ s.data = (char *) data;
+ s.len = len;
+
+ /*
+ * We use first byte of message for additional communication between
+ * main Logical replication worker and Apply BGWorkers, so if it
+ * differs from 'w', then process it first.
+ */
+ switch (pq_getmsgbyte(&s))
+ {
+ /* Stream stop */
+ case 'E':
+ {
+ in_remote_transaction = false;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ SpinLockRelease(&pst->mutex);
+ SetLatch(®istrant->procLatch);
+
+ elog(LOG, "[Apply BGW #%u] ended processing streaming chunk, waiting on shm_mq_receive", pst->n);
+
+ continue;
+ }
+ /* Reassign to the new transaction */
+ case 'R':
+ {
+ elog(LOG, "[Apply BGW #%u] switching from processing xid %u to xid %u",
+ pst->n, stream_xid, pst->stream_xid);
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ continue;
+ }
+ /* Finished processing xact */
+ case 'F':
+ {
+ elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid);
+
+ MemoryContextSwitchTo(ApplyContext);
+
+ CommitTransactionCommand();
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ SpinLockRelease(&pst->mutex);
+
+ continue;
+ }
+ default:
+ break;
+ }
+
+ pq_getmsgint64(&s); // Read LSN info
+ pq_getmsgint64(&s); // TODO Do we need to process it here again somehow?
+ pq_getmsgint64(&s);
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ elog(DEBUG5, "[Apply BGW #%u] applying dispatch for action=%s",
+ pst->n, (char *) &s.data[s.cursor]);
+ apply_dispatch(&s);
+ }
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextReset(ApplyMessageContext);
+ }
+
+ CommitTransactionCommand();
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(ApplyContext);
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ // if (pst->workers_finished == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+
+ elog(LOG, "[Apply BGW #%u] exiting", pst->n);
+
+ /* Signal main process that we are done. */
+ // ConditionVariableBroadcast(&cv);
+ SetLatch(®istrant->procLatch);
+
+ /*
+ * We're done. Explicitly detach the shared memory segment so that we
+ * don't get a resource leak warning at commit time. This will fire any
+ * on_dsm_detach callbacks we've registered, as well. Once that's done,
+ * we can go ahead and exit.
+ */
+ dsm_detach(seg);
+ proc_exit(0);
+}
+
+/*
+ * When we receive a SIGTERM, we set InterruptPending and ProcDiePending just
+ * like a normal backend. The next CHECK_FOR_INTERRUPTS() will do the right
+ * thing.
+ */
+static void
+handle_sigterm(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ SetLatch(MyLatch);
+
+ if (!proc_exit_inprogress)
+ {
+ InterruptPending = true;
+ ProcDiePending = true;
+ }
+
+ errno = save_errno;
+}
+
+/*
+ * Set up a dynamic shared memory segment.
+ *
+ * We set up a control region that contains a ParallelState,
+ * plus one region per message queue. There are as many message queues as
+ * the number of workers.
+ */
+static void
+setup_dsm(WorkerState *wstate)
+{
+ shm_toc_estimator e;
+ int toc_key = 0;
+ Size segsize;
+ dsm_segment *seg;
+ shm_toc *toc;
+ ParallelState *pst;
+ shm_mq *mq;
+ int64 queue_size = 160000000; /* 16 MB for now */
+
+ /* Ensure a valid queue size. */
+ if (queue_size < 0 || ((uint64) queue_size) < shm_mq_minimum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size must be at least %zu bytes",
+ shm_mq_minimum_size)));
+ if (queue_size != ((Size) queue_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size overflows size_t")));
+
+ /*
+ * Estimate how much shared memory we need.
+ *
+ * Because the TOC machinery may choose to insert padding of oddly-sized
+ * requests, we must estimate each chunk separately.
+ *
+ * We need one key to register the location of the header, and we need
+ * nworkers keys to track the locations of the message queues.
+ */
+ shm_toc_initialize_estimator(&e);
+ shm_toc_estimate_chunk(&e, sizeof(ParallelState));
+ shm_toc_estimate_chunk(&e, (Size) queue_size);
+
+ shm_toc_estimate_keys(&e, 1 + 1);
+ segsize = shm_toc_estimate(&e);
+
+ /* Create the shared memory segment and establish a table of contents. */
+ seg = dsm_create(shm_toc_estimate(&e), 0);
+ toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg),
+ segsize);
+
+ /* Set up the header region. */
+ pst = shm_toc_allocate(toc, sizeof(ParallelState));
+ SpinLockInit(&pst->mutex);
+ pst->attached = false;
+ pst->ready = false;
+ pst->finished = false;
+ pst->database_id = MyDatabaseId;
+ pst->subid = MyLogicalRepWorker->subid;
+ pst->stream_xid = stream_xid;
+ pst->authenticated_user_id = GetAuthenticatedUserId();
+ pst->n = nworkers + 1;
+ // ConditionVariableInit(&pst->cv);
+
+ shm_toc_insert(toc, toc_key++, pst);
+
+ /* Set up one message queue per worker, plus one. */
+ mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size),
+ (Size) queue_size);
+ shm_toc_insert(toc, toc_key++, mq);
+ shm_mq_set_sender(mq, MyProc);
+
+ /* Attach the queues. */
+ wstate->mq_handle = shm_mq_attach(mq, seg, wstate->handle);
+
+ /* Return results to caller. */
+ wstate->dsm_seg = seg;
+ wstate->pstate = pst;
+}
+
+/*
+ * Register background workers.
+ */
+static void
+setup_background_worker(WorkerState *wstate)
+{
+ MemoryContext oldcontext;
+ BackgroundWorker worker;
+
+ elog(LOG, "setting up apply worker #%u", nworkers + 1);
+
+ /*
+ * TOCHECK: We need the worker_state object and the background worker handles to
+ * which it points to be allocated in TopMemoryContext rather than
+ * ApplyMessageContext; otherwise, they'll be destroyed before the on_dsm_detach
+ * hooks run.
+ */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+
+ setup_dsm(wstate);
+
+ /*
+ * Arrange to kill all the workers if we abort before all workers are
+ * finished hooking themselves up to the dynamic shared memory segment.
+ *
+ * If we die after all the workers have finished hooking themselves up to
+ * the dynamic shared memory segment, we'll mark the two queues to which
+ * we're directly connected as detached, and the worker(s) connected to
+ * those queues will exit, marking any other queues to which they are
+ * connected as detached. This will cause any as-yet-unaware workers
+ * connected to those queues to exit in their turn, and so on, until
+ * everybody exits.
+ *
+ * But suppose the workers which are supposed to connect to the queues to
+ * which we're directly attached exit due to some error before they
+ * actually attach the queues. The remaining workers will have no way of
+ * knowing this. From their perspective, they're still waiting for those
+ * workers to start, when in fact they've already died.
+ */
+ on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ /* Configure a worker. */
+ MemSet(&worker, 0, sizeof(BackgroundWorker));
+
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_ConsistentState;
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_notify_pid = MyProcPid;
+ sprintf(worker.bgw_library_name, "postgres");
+ sprintf(worker.bgw_function_name, "LogicalApplyBgwMain");
+
+ worker.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(wstate->dsm_seg));
+
+ /* Register the workers. */
+ snprintf(worker.bgw_name, BGW_MAXLEN,
+ "logical replication apply worker #%u for subscription %u",
+ nworkers + 1, MySubscription->oid);
+ if (!RegisterDynamicBackgroundWorker(&worker, &wstate->handle))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not register background process"),
+ errhint("You may need to increase max_worker_processes.")));
+
+ /* All done. */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Wait for worker to become ready. */
+ wait_for_worker(wstate);
+
+ /*
+ * Once we reach this point, all workers are ready. We no longer need to
+ * kill them if we die; they'll die on their own as the message queues
+ * shut down.
+ */
+ cancel_on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ nworkers += 1;
+}
+
+static void
+cleanup_background_worker(dsm_segment *seg, Datum arg)
+{
+ WorkerState *wstate = (WorkerState *) DatumGetPointer(arg);
+
+ TerminateBackgroundWorker(wstate->handle);
+}
+
+static void
+wait_for_worker(WorkerState *wstate)
+{
+ bool result = false;
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool ready;
+
+ /* If the worker is ready, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ ready = wstate->pstate->ready;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (ready)
+ {
+ result = true;
+ break;
+ }
+
+ /* If any workers (or the postmaster) have died, we have failed. */
+ if (!check_worker_status(wstate))
+ {
+ result = false;
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ // ConditionVariableCancelSleep();
+
+ if (!result)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("one or more background workers failed to start")));
+}
+
+static bool
+check_worker_status(WorkerState *wstate)
+{
+ BgwHandleStatus status;
+ pid_t pid;
+
+ status = GetBackgroundWorkerPid(wstate->handle, &pid);
+ if (status == BGWH_STOPPED || status == BGWH_POSTMASTER_DIED)
+ return false;
+
+ /* Otherwise, things still look OK. */
+ return true;
+}
+
+static void
+wait_for_worker_to_finish(WorkerState *wstate)
+{
+ elog(LOG, "waiting for apply worker #%u to finish processing xid %u",
+ wstate->pstate->n, wstate->xid);
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool finished;
+
+ /* If the worker is finished, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ finished = wstate->pstate->finished;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (finished)
+ {
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index bc45194..342e9fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -839,6 +839,7 @@ typedef enum
WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATING,
WAIT_EVENT_HASH_GROW_BUCKETS_ELECTING,
WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY,
WAIT_EVENT_LOGICAL_SYNC_DATA,
WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
WAIT_EVENT_MQ_INTERNAL,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index bf02cbc..479c2e2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -122,12 +122,10 @@ extern TransactionId logicalrep_read_stream_stop(StringInfo in);
extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn);
-extern TransactionId logicalrep_read_stream_commit(StringInfo out,
- LogicalRepCommitData *commit_data);
+extern void logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
TransactionId subxid);
-extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid);
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index e9524ae..30ad402 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -13,6 +13,7 @@
#define LOGICALWORKER_H
extern void ApplyWorkerMain(Datum main_arg);
+extern void LogicalApplyBgwMain(Datum main_arg);
extern bool IsLogicalWorker(void);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-03 09:13 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-03 09:13 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
> >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com
> >
> >wrote:
> >
> >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >> >
> >> >On further testing, I found that the patch seems to have problems with
> >> >toast. Consider below scenario:
> >> >Session-1
> >> >Create table large_text(t1 text);
> >> >INSERT INTO large_text
> >> >SELECT (SELECT string_agg('x', ',')
> >> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >> >
> >> >Session-2
> >> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >> >'test_decoding');
> >> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL);
> >> >*--kaboom*
> >> >
> >> >The second statement in Session-2 leads to a crash.
> >> >
> >>
> >> OK, thanks for the report - will investigate.
> >>
> >
> >It was an assertion failure in ReorderBufferCleanupTXN at below line:
> >+ /* Check we're not mixing changes from different transactions. */
> >+ Assert(change->txn == txn);
> >
>
> Can you still reproduce this issue with the patch I sent on 28/9? I have
> been unable to trigger the failure, and it seems pretty similar to the
> failure you reported (and I fixed) on 28/9.
>
Yes, it seems we need a similar change in ReorderBufferAddNewTupleCids. I
think in session-2 you need to create replication slot before creating
table in session-1 to see this problem.
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2196,6 +2196,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb,
TransactionId xid,
change->data.tuplecid.cmax = cmax;
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
+ change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
dlist_push_tail(&txn->tuplecids, &change->node);
Few more comments:
-----------------------------------
1.
+static bool
+check_logical_decoding_work_mem(int *newval, void **extra, GucSource
source)
+{
+ /*
+ * -1 indicates fallback.
+ *
+ * If we haven't yet changed the boot_val default of -1, just let it be.
+ * logical decoding will look to maintenance_work_mem instead.
+ */
+ if (*newval == -1)
+ return true;
+
+ /*
+ * We clamp manually-set values to at least 64kB. The maintenance_work_mem
+ * uses a higher minimum value (1MB), so this is OK.
+ */
+ if (*newval < 64)
+ *newval = 64;
I think this needs to be changed as now we don't rely on
maintenance_work_mem. Another thing related to this is that I think the
default value for logical_decoding_work_mem still seems to be -1. We need
to make it to 64MB. I have seen this while debugging memory accounting
changes. I think this is the reason why I was not seeing toast related
changes being serialized because, in that test, I haven't changed the
default value of logical_decoding_work_mem.
2.
+ /*
+ * We're going modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
/going modify/going to modify/
3.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb,
ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
It is not very clear why this change is required. Basically, this is done
at commit time after which actually we shouldn't attempt to spill these
changes. This is mentioned in comments as well, but it is not clear if
that is the case, then how and when accounting can create a problem. If
possible, can you explain it with an example?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-10 10:27 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-10 10:27 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have attempted to test the performance of (Stream + Spill) vs
> (Stream + BGW pool) and I can see the similar gain what Alexey had
> shown[1].
>
> In addition to this, I have rebased the latest patchset [2] without
> the two-phase logical decoding patch set.
>
> Test results:
> I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> here is my result
> Stream + Spill
> N time on master(sec) Total xact time (sec)
> 1kk 6 21
> 3kk 18 55
>
> Stream + BGW pool
> N time on master(sec) Total xact time (sec)
> 1kk 6 13
> 3kk 19 35
>
> Patch details:
> All the patches are the same as posted on [2] except
> 1. 0006-Gracefully-handle-concurrent-aborts-of-uncommitted -> I have
> removed the handling of error which is specific for 2PC
Here[1], I mentioned that I have removed the 2PC changes from
this[0006] patch but mistakenly I attached the original patch itself
instead of the modified version. So attaching the modified version of
only this patch other patches are the same.
> 2. 0007-Implement-streaming-mode-in-ReorderBuffer -> Rebased without 2PC
> 3. 0009-Extend-the-concurrent-abort-handling-for-in-progress -> New
> patch to handle concurrent abort error for the in-progress transaction
> and also add handling for the sub transaction's abort.
> 4. v3-0014-BGWorkers-pool-for-streamed-transactions-apply -> Rebased
> Alexey's patch
[1] https://www.postgresql.org/message-id/CAFiTN-vHoksqvV4BZ0479NhugGe4QHq_ezngNdDd-YRQ_2cwug%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0006-Gracefully-handle-concurrent-aborts-of-uncommitted-t.patch (application/octet-stream, 12.4K) download
From 9739d48a979868b912f6e1f90d4702808e061a35 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:00:49 +0530
Subject: [PATCH 06/14] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 119 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index e954482..6ce7878 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index ca4b904..3143479 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -679,7 +679,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1529,7 +1529,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1780,7 +1780,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1800,7 +1800,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Every time the CommandId is incremented, we could
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-13 06:54 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-13 06:54 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 2:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
>> >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
>> >wrote:
>> >
>> >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
>> >> >
>> >> >On further testing, I found that the patch seems to have problems with
>> >> >toast. Consider below scenario:
>> >> >Session-1
>> >> >Create table large_text(t1 text);
>> >> >INSERT INTO large_text
>> >> >SELECT (SELECT string_agg('x', ',')
>> >> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>> >> >
>> >> >Session-2
>> >> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
>> >> >'test_decoding');
>> >> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
>> >> >*--kaboom*
>> >> >
>> >> >The second statement in Session-2 leads to a crash.
>> >> >
>> >>
>> >> OK, thanks for the report - will investigate.
>> >>
>> >
>> >It was an assertion failure in ReorderBufferCleanupTXN at below line:
>> >+ /* Check we're not mixing changes from different transactions. */
>> >+ Assert(change->txn == txn);
>> >
>>
>> Can you still reproduce this issue with the patch I sent on 28/9? I have
>> been unable to trigger the failure, and it seems pretty similar to the
>> failure you reported (and I fixed) on 28/9.
>
>
> Yes, it seems we need a similar change in ReorderBufferAddNewTupleCids. I think in session-2 you need to create replication slot before creating table in session-1 to see this problem.
>
> --- a/src/backend/replication/logical/reorderbuffer.c
> +++ b/src/backend/replication/logical/reorderbuffer.c
> @@ -2196,6 +2196,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
> change->data.tuplecid.cmax = cmax;
> change->data.tuplecid.combocid = combocid;
> change->lsn = lsn;
> + change->txn = txn;
> change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
> dlist_push_tail(&txn->tuplecids, &change->node);
>
> Few more comments:
> -----------------------------------
> 1.
> +static bool
> +check_logical_decoding_work_mem(int *newval, void **extra, GucSource source)
> +{
> + /*
> + * -1 indicates fallback.
> + *
> + * If we haven't yet changed the boot_val default of -1, just let it be.
> + * logical decoding will look to maintenance_work_mem instead.
> + */
> + if (*newval == -1)
> + return true;
> +
> + /*
> + * We clamp manually-set values to at least 64kB. The maintenance_work_mem
> + * uses a higher minimum value (1MB), so this is OK.
> + */
> + if (*newval < 64)
> + *newval = 64;
>
> I think this needs to be changed as now we don't rely on maintenance_work_mem. Another thing related to this is that I think the default value for logical_decoding_work_mem still seems to be -1. We need to make it to 64MB. I have seen this while debugging memory accounting changes. I think this is the reason why I was not seeing toast related changes being serialized because, in that test, I haven't changed the default value of logical_decoding_work_mem.
>
> 2.
> + /*
> + * We're going modify the size of the change, so to make sure the
> + * accounting is correct we'll make it look like we're removing the
> + * change now (with the old size), and then re-add it at the end.
> + */
>
>
> /going modify/going to modify/
>
> 3.
> + *
> + * While updating the existing change with detoasted tuple data, we need to
> + * update the memory accounting info, because the change size will differ.
> + * Otherwise the accounting may get out of sync, triggering serialization
> + * at unexpected times.
> + *
> + * We simply subtract size of the change before rejiggering the tuple, and
> + * then adding the new size. This makes it look like the change was removed
> + * and then added back, except it only tweaks the accounting info.
> + *
> + * In particular it can't trigger serialization, which would be pointless
> + * anyway as it happens during commit processing right before handing
> + * the change to the output plugin.
> */
> static void
> ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
> @@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
> if (txn->toast_hash == NULL)
> return;
>
> + /*
> + * We're going modify the size of the change, so to make sure the
> + * accounting is correct we'll make it look like we're removing the
> + * change now (with the old size), and then re-add it at the end.
> + */
> + ReorderBufferChangeMemoryUpdate(rb, change, false);
>
> It is not very clear why this change is required. Basically, this is done at commit time after which actually we shouldn't attempt to spill these changes. This is mentioned in comments as well, but it is not clear if that is the case, then how and when accounting can create a problem. If possible, can you explain it with an example?
>
IIUC, we are keeping the track of the memory in ReorderBuffer which is
common across the transactions. So even if this transaction is
committing and will not spill to dis but we need to keep the memory
accounting correct for the future changes in other transactions.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-13 11:46 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-13 11:46 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Oct 13, 2019 at 12:25 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 2:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3.
> > + *
> > + * While updating the existing change with detoasted tuple data, we need to
> > + * update the memory accounting info, because the change size will differ.
> > + * Otherwise the accounting may get out of sync, triggering serialization
> > + * at unexpected times.
> > + *
> > + * We simply subtract size of the change before rejiggering the tuple, and
> > + * then adding the new size. This makes it look like the change was removed
> > + * and then added back, except it only tweaks the accounting info.
> > + *
> > + * In particular it can't trigger serialization, which would be pointless
> > + * anyway as it happens during commit processing right before handing
> > + * the change to the output plugin.
> > */
> > static void
> > ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
> > @@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
> > if (txn->toast_hash == NULL)
> > return;
> >
> > + /*
> > + * We're going modify the size of the change, so to make sure the
> > + * accounting is correct we'll make it look like we're removing the
> > + * change now (with the old size), and then re-add it at the end.
> > + */
> > + ReorderBufferChangeMemoryUpdate(rb, change, false);
> >
> > It is not very clear why this change is required. Basically, this is done at commit time after which actually we shouldn't attempt to spill these changes. This is mentioned in comments as well, but it is not clear if that is the case, then how and when accounting can create a problem. If possible, can you explain it with an example?
> >
> IIUC, we are keeping the track of the memory in ReorderBuffer which is
> common across the transactions. So even if this transaction is
> committing and will not spill to dis but we need to keep the memory
> accounting correct for the future changes in other transactions.
>
You are right. I somehow missed that we need to keep the size
computation in sync even during commit for other in-progress
transactions in the ReorderBuffer. You can ignore this point or maybe
slightly adjust the comment to make it explicit.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-14 01:21 Craig Ringer <craig@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Craig Ringer @ 2019-10-14 01:21 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, 13 Oct 2019 at 19:50, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Oct 13, 2019 at 12:25 PM Dilip Kumar <dilipbalaut@gmail.com>
> wrote:
> >
> > On Thu, Oct 3, 2019 at 2:43 PM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > >
> > > 3.
> > > + *
> > > + * While updating the existing change with detoasted tuple data, we
> need to
> > > + * update the memory accounting info, because the change size will
> differ.
> > > + * Otherwise the accounting may get out of sync, triggering
> serialization
> > > + * at unexpected times.
> > > + *
> > > + * We simply subtract size of the change before rejiggering the
> tuple, and
> > > + * then adding the new size. This makes it look like the change was
> removed
> > > + * and then added back, except it only tweaks the accounting info.
> > > + *
> > > + * In particular it can't trigger serialization, which would be
> pointless
> > > + * anyway as it happens during commit processing right before handing
> > > + * the change to the output plugin.
> > > */
> > > static void
> > > ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
> > > @@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb,
> ReorderBufferTXN *txn,
> > > if (txn->toast_hash == NULL)
> > > return;
> > >
> > > + /*
> > > + * We're going modify the size of the change, so to make sure the
> > > + * accounting is correct we'll make it look like we're removing the
> > > + * change now (with the old size), and then re-add it at the end.
> > > + */
> > > + ReorderBufferChangeMemoryUpdate(rb, change, false);
> > >
> > > It is not very clear why this change is required. Basically, this is
> done at commit time after which actually we shouldn't attempt to spill
> these changes. This is mentioned in comments as well, but it is not clear
> if that is the case, then how and when accounting can create a problem. If
> possible, can you explain it with an example?
> > >
> > IIUC, we are keeping the track of the memory in ReorderBuffer which is
> > common across the transactions. So even if this transaction is
> > committing and will not spill to dis but we need to keep the memory
> > accounting correct for the future changes in other transactions.
> >
>
> You are right. I somehow missed that we need to keep the size
> computation in sync even during commit for other in-progress
> transactions in the ReorderBuffer. You can ignore this point or maybe
> slightly adjust the comment to make it explicit.
Does anyone object if we add the reorder buffer total size & in-memory size
to struct WalSnd too, so we can report it in pg_stat_replication?
I can follow up with a patch to add on top of this one if you think it's
reasonable. I'll also take the opportunity to add a number of tracepoints
across the walsender and logical decoding, since right now it's very opaque
in production systems ... and everyone just LOVES hunting down debug syms
and attaching gdb to production DBs.
--
Craig Ringer http://www.2ndQuadrant.com/
2ndQuadrant - PostgreSQL Solutions for the Enterprise
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-14 02:11 Amit Kapila <amit.kapila16@gmail.com>
parent: Craig Ringer <craig@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-10-14 02:11 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Oct 14, 2019 at 6:51 AM Craig Ringer <craig@2ndquadrant.com> wrote:
>
> On Sun, 13 Oct 2019 at 19:50, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>
>
> Does anyone object if we add the reorder buffer total size & in-memory size to struct WalSnd too, so we can report it in pg_stat_replication?
>
There is already a patch
(0011-Track-statistics-for-streaming-spilling) in this series posted
by Tomas[1] which tracks important statistics in WalSnd which I think
are good enough. Have you checked that? I am not sure if adding
additional size will help, but I might be missing something.
> I can follow up with a patch to add on top of this one if you think it's reasonable. I'll also take the opportunity to add a number of tracepoints across the walsender and logical decoding, since right now it's very opaque in production systems ... and everyone just LOVES hunting down debug syms and attaching gdb to production DBs.
>
Sure, adding tracepoints can be helpful, but isn't it better to start
that as a separate thread?
[1] - https://www.postgresql.org/message-id/20190928190917.hrpknmq76v3ts3lj%40development
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-14 09:39 Dilip Kumar <dilipbalaut@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-14 09:39 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
> >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
> >wrote:
> >
> >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >> >
> >> >On further testing, I found that the patch seems to have problems with
> >> >toast. Consider below scenario:
> >> >Session-1
> >> >Create table large_text(t1 text);
> >> >INSERT INTO large_text
> >> >SELECT (SELECT string_agg('x', ',')
> >> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >> >
> >> >Session-2
> >> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >> >'test_decoding');
> >> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
> >> >*--kaboom*
> >> >
> >> >The second statement in Session-2 leads to a crash.
> >> >
> >>
> >> OK, thanks for the report - will investigate.
> >>
> >
> >It was an assertion failure in ReorderBufferCleanupTXN at below line:
> >+ /* Check we're not mixing changes from different transactions. */
> >+ Assert(change->txn == txn);
> >
>
> Can you still reproduce this issue with the patch I sent on 28/9? I have
> been unable to trigger the failure, and it seems pretty similar to the
> failure you reported (and I fixed) on 28/9.
>
> >> >Other than that, I am not sure if the changes related to spill to disk
> >> >after logical_decoding_work_mem works for toast table as I couldn't hit
> >> >that code for toast table case, but I might be missing something. As
> >> >mentioned previously, I feel there should be some way to test whether this
> >> >patch works for the cases it claims to work. As of now, I have to check
> >> >via debugging. Let me know if there is any way, I can test this.
> >> >
> >>
> >> That's one of the reasons why I proposed to move the statistics (which
> >> say how many transactions / bytes were spilled to disk) from a later
> >> patch in the series. I don't think there's a better way.
> >>
> >>
> >I like that idea, but I think you need to split that patch to only get the
> >stats related to the spill. It would be easier to review if you can
> >prepare that atop of
> >0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
> >
>
> Sure, I wasn't really proposing to adding all stats from that patch,
> including those related to streaming. We need to extract just those
> related to spilling. And yes, it needs to be moved right after 0001.
>
I have extracted the spilling related code to a separate patch on top
of 0001. I have also fixed some bugs and review comments and attached
as a separate patch. Later I can merge it to the main patch if you
agree with the changes.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch (application/octet-stream, 33.6K) download
From 4bfca35b149a303779bee49d96e3be25b914478f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:04:54 +0200
Subject: [PATCH 1/2] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to trigger very aggressive streaming. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem may be set either in postgresql.conf,
in which case it serves as the default for all publishers on that
instance, or when creating the subscription, using a work_mem
paramemter in the WITH clause (specifies number of kilobytes).
---
doc/src/sgml/config.sgml | 21 ++
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/reorderbuffer.c | 292 ++++++++++++++++++++-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++-
src/backend/utils/misc/guc.c | 36 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/reorderbuffer.h | 16 ++
src/include/replication/walreceiver.h | 1 +
13 files changed, 441 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 47b12c6..f1d13a0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1716,6 +1716,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee283..7e3ba8e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e67a58..d85e831 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +98,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -182,6 +185,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +338,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -341,7 +356,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -419,6 +434,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -682,10 +703,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -710,6 +734,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -721,7 +752,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -759,7 +791,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -796,7 +828,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a..65b3266 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8ce28ad..6228140 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is then
+ * serialized to disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when
+ * reaching memory limit. So we just don't count them, because it
+ * might easily trigger a pointless attempt to spill/stream.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2230,20 +2323,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2670,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3019,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters).
+ * And we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3248,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3437,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 11e6331..f737afb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1725,6 +1725,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757..317c5d4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/inval.h"
#include "utils/int8.h"
#include "utils/memutils.h"
@@ -90,11 +91,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -140,6 +142,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +199,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 2178e1c..5d7e687 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -65,6 +65,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
+#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
@@ -191,6 +192,7 @@ static bool check_maxconnections(int *newval, void **extra, GucSource source);
static bool check_max_worker_processes(int *newval, void **extra, GucSource source);
static bool check_autovacuum_max_workers(int *newval, void **extra, GucSource source);
static bool check_max_wal_senders(int *newval, void **extra, GucSource source);
+static bool check_logical_decoding_work_mem(int *newval, void **extra, GucSource source);
static bool check_autovacuum_work_mem(int *newval, void **extra, GucSource source);
static bool check_effective_io_concurrency(int *newval, void **extra, GucSource source);
static void assign_effective_io_concurrency(int newval, void *extra);
@@ -2251,6 +2253,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk or streaming."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ -1, -1, MAX_KILOBYTES,
+ check_logical_decoding_work_mem, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -11286,6 +11300,28 @@ check_max_wal_senders(int *newval, void **extra, GucSource source)
}
static bool
+check_logical_decoding_work_mem(int *newval, void **extra, GucSource source)
+{
+ /*
+ * -1 indicates fallback.
+ *
+ * If we haven't yet changed the boot_val default of -1, just let it be.
+ * logical decoding will look to maintenance_work_mem instead.
+ */
+ if (*newval == -1)
+ return true;
+
+ /*
+ * We clamp manually-set values to at least 64kB. The maintenance_work_mem
+ * uses a higher minimum value (1MB), so this is OK.
+ */
+ if (*newval < 64)
+ *newval = 64;
+
+ return true;
+}
+
+static bool
check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
{
/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3..00a22b8 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..4dcef80 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
bugs_and_review_comments_fix.patch (application/octet-stream, 3.7K) download
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e9d57b4..5c69359 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2200,6 +2200,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.cmax = cmax;
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
+ change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
dlist_push_tail(&txn->tuplecids, &change->node);
@@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* update the statistics */
rb->spillCount += 1;
- rb->spillTxns += txn->serialized ? 1 : 0;
+ rb->spillTxns += txn->serialized ? 0 : 1;
rb->spillBytes += size;
Assert(spilled == txn->nentries_mem);
@@ -3292,7 +3293,7 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
return;
/*
- * We're going modify the size of the change, so to make sure the
+ * We're going to modify the size of the change, so to make sure the
* accounting is correct we'll make it look like we're removing the
* change now (with the old size), and then re-add it at the end.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5d7e687..c7252cf 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -192,7 +192,6 @@ static bool check_maxconnections(int *newval, void **extra, GucSource source);
static bool check_max_worker_processes(int *newval, void **extra, GucSource source);
static bool check_autovacuum_max_workers(int *newval, void **extra, GucSource source);
static bool check_max_wal_senders(int *newval, void **extra, GucSource source);
-static bool check_logical_decoding_work_mem(int *newval, void **extra, GucSource source);
static bool check_autovacuum_work_mem(int *newval, void **extra, GucSource source);
static bool check_effective_io_concurrency(int *newval, void **extra, GucSource source);
static void assign_effective_io_concurrency(int newval, void *extra);
@@ -2261,8 +2260,8 @@ static struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&logical_decoding_work_mem,
- -1, -1, MAX_KILOBYTES,
- check_logical_decoding_work_mem, NULL, NULL
+ 65536, 64, MAX_KILOBYTES,
+ NULL, NULL, NULL
},
/*
@@ -11300,28 +11299,6 @@ check_max_wal_senders(int *newval, void **extra, GucSource source)
}
static bool
-check_logical_decoding_work_mem(int *newval, void **extra, GucSource source)
-{
- /*
- * -1 indicates fallback.
- *
- * If we haven't yet changed the boot_val default of -1, just let it be.
- * logical decoding will look to maintenance_work_mem instead.
- */
- if (*newval == -1)
- return true;
-
- /*
- * We clamp manually-set values to at least 64kB. The maintenance_work_mem
- * uses a higher minimum value (1MB), so this is OK.
- */
- if (*newval < 64)
- *newval = 64;
-
- return true;
-}
-
-static bool
check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
{
/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 00a22b8..04529aa 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,7 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
-#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
+i#logical_decoding_work_mem = 64MB # min 64kB
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
0002-Track-statistics-for-spilling.patch (application/octet-stream, 11.0K) download
From cc021dc5dba6bf0059595bc29388bb20ce49c405 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 11 Oct 2019 09:07:41 +0530
Subject: [PATCH 2/2] Track statistics for spilling
---
doc/src/sgml/monitoring.sgml | 23 ++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 10 ++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 101 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 828e908..1965a8d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2121,6 +2121,29 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
with security-sensitive fields obfuscated.
</entry>
</row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.
+ </entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a47..2ee2a06 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 6228140..e9d57b4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2414,6 +2418,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2472,6 +2477,11 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillTxns += txn->serialized ? 1 : 0;
+ rb->spillBytes += size;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index eb4a98c..d74f6f8 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2319,6 +2326,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3230,7 +3240,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3285,6 +3295,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3305,6 +3318,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3386,6 +3402,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3622,3 +3643,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4dcef80..ba7f9f0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 210e9cd..750bdc4 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1951,9 +1951,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-18 12:01 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-10-18 12:01 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Oct 14, 2019 at 3:09 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> >
> > Sure, I wasn't really proposing to adding all stats from that patch,
> > including those related to streaming. We need to extract just those
> > related to spilling. And yes, it needs to be moved right after 0001.
> >
> I have extracted the spilling related code to a separate patch on top
> of 0001. I have also fixed some bugs and review comments and attached
> as a separate patch. Later I can merge it to the main patch if you
> agree with the changes.
>
Few comments
-------------------------
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer
1.
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk or streaming."),
+ GUC_UNIT_KB
+ },
I think we can remove 'or streaming' from above sentence for now. We
can add it later with later patch where streaming will be allowed.
2.
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
It is not clear why we need this parameter at least with this patch?
I have raised this multiple times [1][2].
bugs_and_review_comments_fix
1.
},
&logical_decoding_work_mem,
- -1, -1, MAX_KILOBYTES,
- check_logical_decoding_work_mem, NULL, NULL
+ 65536, 64, MAX_KILOBYTES,
+ NULL, NULL, NULL
I think the default value should be 1MB similar to
maintenance_work_mem. The same was true before this change.
2. -#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
maintenance_work_mem
+i#logical_decoding_work_mem = 64MB # min 64kB
It seems the 'i' is a leftover character in the above change. Also,
change the default value considering the previous point.
3.
@@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb,
ReorderBufferTXN *txn)
/* update the statistics */
rb->spillCount += 1;
- rb->spillTxns += txn->serialized ? 1 : 0;
+ rb->spillTxns += txn->serialized ? 0 : 1;
rb->spillBytes += size;
Why is this change required? Shouldn't we increase the spillTxns
count only when the txn is serialized?
0002-Track-statistics-for-spilling
1.
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
The parameter name is wrong here. /logical_work_mem/logical_decoding_work_mem
2.
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.
+ </entry>
+ </row>
In all the above cases, the explanation text starts immediately after
<entry> tag, but the general coding practice is to start from the next
line, see the explanation of nearby parameters.
It seems these parameters are added in pg-stat-wal-receiver-view in
the docs, but in code, it is present as part of pg_stat_replication.
It seems doc needs to be updated. Am, I missing something?
3.
ReorderBufferSerializeTXN()
{
..
/* update the statistics */
rb->spillCount += 1;
rb->spillTxns += txn->serialized ? 0 : 1;
rb->spillBytes += size;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
txn->serialized = true;
..
}
I am not able to understand the above code. We are setting the
serialized parameter a few lines after we check it and increment the
spillTxns count. Can you please explain it?
Also, isn't spillTxns count bit confusing, because in some cases it
will include subtransactions and other cases (where the largest picked
transaction is a subtransaction) it won't include it?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-21 05:18 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-21 05:18 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
I have replied to some of your questions inline. I will work on the
remaining comments and post the patch for the same.
> > >
> > > Sure, I wasn't really proposing to adding all stats from that patch,
> > > including those related to streaming. We need to extract just those
> > > related to spilling. And yes, it needs to be moved right after 0001.
> > >
> > I have extracted the spilling related code to a separate patch on top
> > of 0001. I have also fixed some bugs and review comments and attached
> > as a separate patch. Later I can merge it to the main patch if you
> > agree with the changes.
> >
>
> Few comments
> -------------------------
> 0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer
> 1.
> + {
> + {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
> + gettext_noop("Sets the maximum memory to be used for logical decoding."),
> + gettext_noop("This much memory can be used by each internal "
> + "reorder buffer before spilling to disk or streaming."),
> + GUC_UNIT_KB
> + },
>
> I think we can remove 'or streaming' from above sentence for now. We
> can add it later with later patch where streaming will be allowed.
>
> 2.
> @@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
> </para>
> </listitem>
> </varlistentry>
> +
> + <varlistentry>
> + <term><literal>work_mem</literal> (<type>integer</type>)</term>
> + <listitem>
> + <para>
> + Limits the amount of memory used to decode changes on the
> + publisher. If not specified, the publisher will use the default
> + specified by <varname>logical_decoding_work_mem</varname>. When
> + needed, additional data are spilled to disk.
> + </para>
> + </listitem>
> + </varlistentry>
>
> It is not clear why we need this parameter at least with this patch?
> I have raised this multiple times [1][2].
>
> bugs_and_review_comments_fix
> 1.
> },
> &logical_decoding_work_mem,
> - -1, -1, MAX_KILOBYTES,
> - check_logical_decoding_work_mem, NULL, NULL
> + 65536, 64, MAX_KILOBYTES,
> + NULL, NULL, NULL
>
> I think the default value should be 1MB similar to
> maintenance_work_mem. The same was true before this change.
>
> 2. -#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
> maintenance_work_mem
> +i#logical_decoding_work_mem = 64MB # min 64kB
>
> It seems the 'i' is a leftover character in the above change. Also,
> change the default value considering the previous point.
>
> 3.
> @@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb,
> ReorderBufferTXN *txn)
>
> /* update the statistics */
> rb->spillCount += 1;
> - rb->spillTxns += txn->serialized ? 1 : 0;
> + rb->spillTxns += txn->serialized ? 0 : 1;
> rb->spillBytes += size;
>
> Why is this change required? Shouldn't we increase the spillTxns
> count only when the txn is serialized?
Prior to this change it was increasing the rb->spillTxns, every time
we try to serialize the changes of the transaction. Now, only we
increase first time when it is not yet serialized.
> 0002-Track-statistics-for-spilling
> 1.
> + <row>
> + <entry><structfield>spill_txns</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of transactions spilled to disk after the memory used by
> + logical decoding exceeds <literal>logical_work_mem</literal>. The
> + counter gets incremented both for toplevel transactions and
> + subtransactions.
> + </entry>
> + </row>
>
> The parameter name is wrong here. /logical_work_mem/logical_decoding_work_mem
>
> 2.
> + <row>
> + <entry><structfield>spill_txns</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of transactions spilled to disk after the memory used by
> + logical decoding exceeds <literal>logical_work_mem</literal>. The
> + counter gets incremented both for toplevel transactions and
> + subtransactions.
> + </entry>
> + </row>
> + <row>
> + <entry><structfield>spill_count</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of times transactions were spilled to disk. Transactions
> + may get spilled repeatedly, and this counter gets incremented on every
> + such invocation.
> + </entry>
> + </row>
> + <row>
> + <entry><structfield>spill_bytes</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Amount of decoded transaction data spilled to disk.
> + </entry>
> + </row>
>
> In all the above cases, the explanation text starts immediately after
> <entry> tag, but the general coding practice is to start from the next
> line, see the explanation of nearby parameters.
>
> It seems these parameters are added in pg-stat-wal-receiver-view in
> the docs, but in code, it is present as part of pg_stat_replication.
> It seems doc needs to be updated. Am, I missing something?
>
> 3.
> ReorderBufferSerializeTXN()
> {
> ..
> /* update the statistics */
> rb->spillCount += 1;
> rb->spillTxns += txn->serialized ? 0 : 1;
> rb->spillBytes += size;
>
> Assert(spilled == txn->nentries_mem);
> Assert(dlist_is_empty(&txn->changes));
> txn->nentries_mem = 0;
> txn->serialized = true;
> ..
> }
>
> I am not able to understand the above code. We are setting the
> serialized parameter a few lines after we check it and increment the
> spillTxns count. Can you please explain it?
Basically, when the first time we attempt to serialize a transaction,
txn->serialized will be false, that time we will increment the
rb->spillTxns and after that set txn->serialized to true. From next
time onwards if we try to serialize the same transaction we will not
increment the rb->spillTxns so that we count each transaction only
once.
>
> Also, isn't spillTxns count bit confusing, because in some cases it
> will include subtransactions and other cases (where the largest picked
> transaction is a subtransaction) it won't include it?
I did not understand your comment completely. Basically, every
transaction which we are serializing we will increase the count first
time right? whether it is the main transaction or the sub-transaction.
Am I missing something?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-21 09:20 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-21 09:20 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Oct 21, 2019 at 10:48 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > 3.
> > @@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb,
> > ReorderBufferTXN *txn)
> >
> > /* update the statistics */
> > rb->spillCount += 1;
> > - rb->spillTxns += txn->serialized ? 1 : 0;
> > + rb->spillTxns += txn->serialized ? 0 : 1;
> > rb->spillBytes += size;
> >
> > Why is this change required? Shouldn't we increase the spillTxns
> > count only when the txn is serialized?
>
> Prior to this change it was increasing the rb->spillTxns, every time
> we try to serialize the changes of the transaction. Now, only we
> increase first time when it is not yet serialized.
>
> >
> > 3.
> > ReorderBufferSerializeTXN()
> > {
> > ..
> > /* update the statistics */
> > rb->spillCount += 1;
> > rb->spillTxns += txn->serialized ? 0 : 1;
> > rb->spillBytes += size;
> >
> > Assert(spilled == txn->nentries_mem);
> > Assert(dlist_is_empty(&txn->changes));
> > txn->nentries_mem = 0;
> > txn->serialized = true;
> > ..
> > }
> >
> > I am not able to understand the above code. We are setting the
> > serialized parameter a few lines after we check it and increment the
> > spillTxns count. Can you please explain it?
>
> Basically, when the first time we attempt to serialize a transaction,
> txn->serialized will be false, that time we will increment the
> rb->spillTxns and after that set txn->serialized to true. From next
> time onwards if we try to serialize the same transaction we will not
> increment the rb->spillTxns so that we count each transaction only
> once.
>
Your explanation for both the above comments makes sense to me. Can
you please add some comments along these lines because it is not
apparent why one wants to increase the spillTxns counter when
txn->serialized is false?
> >
> > Also, isn't spillTxns count bit confusing, because in some cases it
> > will include subtransactions and other cases (where the largest picked
> > transaction is a subtransaction) it won't include it?
>
> I did not understand your comment completely. Basically, every
> transaction which we are serializing we will increase the count first
> time right? whether it is the main transaction or the sub-transaction.
>
It was not clear to me earlier whether we always increase the
spillTxns counter for subtransactions or not. But now, looking at
code carefully, it is clear that is it is getting increased in every
case. In short, you don't need to do anything for this comment.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-21 09:41 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-21 09:41 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Oct 21, 2019 at 2:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Oct 21, 2019 at 10:48 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > 3.
> > > @@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb,
> > > ReorderBufferTXN *txn)
> > >
> > > /* update the statistics */
> > > rb->spillCount += 1;
> > > - rb->spillTxns += txn->serialized ? 1 : 0;
> > > + rb->spillTxns += txn->serialized ? 0 : 1;
> > > rb->spillBytes += size;
> > >
> > > Why is this change required? Shouldn't we increase the spillTxns
> > > count only when the txn is serialized?
> >
> > Prior to this change it was increasing the rb->spillTxns, every time
> > we try to serialize the changes of the transaction. Now, only we
> > increase first time when it is not yet serialized.
> >
> > >
> > > 3.
> > > ReorderBufferSerializeTXN()
> > > {
> > > ..
> > > /* update the statistics */
> > > rb->spillCount += 1;
> > > rb->spillTxns += txn->serialized ? 0 : 1;
> > > rb->spillBytes += size;
> > >
> > > Assert(spilled == txn->nentries_mem);
> > > Assert(dlist_is_empty(&txn->changes));
> > > txn->nentries_mem = 0;
> > > txn->serialized = true;
> > > ..
> > > }
> > >
> > > I am not able to understand the above code. We are setting the
> > > serialized parameter a few lines after we check it and increment the
> > > spillTxns count. Can you please explain it?
> >
> > Basically, when the first time we attempt to serialize a transaction,
> > txn->serialized will be false, that time we will increment the
> > rb->spillTxns and after that set txn->serialized to true. From next
> > time onwards if we try to serialize the same transaction we will not
> > increment the rb->spillTxns so that we count each transaction only
> > once.
> >
>
> Your explanation for both the above comments makes sense to me. Can
> you please add some comments along these lines because it is not
> apparent why one wants to increase the spillTxns counter when
> txn->serialized is false?
Ok, I will add comments in the next patch.
>
> > >
> > > Also, isn't spillTxns count bit confusing, because in some cases it
> > > will include subtransactions and other cases (where the largest picked
> > > transaction is a subtransaction) it won't include it?
> >
> > I did not understand your comment completely. Basically, every
> > transaction which we are serializing we will increase the count first
> > time right? whether it is the main transaction or the sub-transaction.
> >
>
> It was not clear to me earlier whether we always increase the
> spillTxns counter for subtransactions or not. But now, looking at
> code carefully, it is clear that is it is getting increased in every
> case. In short, you don't need to do anything for this comment.
ok
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 05:00 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-22 05:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Oct 14, 2019 at 3:09 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > >
> > > Sure, I wasn't really proposing to adding all stats from that patch,
> > > including those related to streaming. We need to extract just those
> > > related to spilling. And yes, it needs to be moved right after 0001.
> > >
> > I have extracted the spilling related code to a separate patch on top
> > of 0001. I have also fixed some bugs and review comments and attached
> > as a separate patch. Later I can merge it to the main patch if you
> > agree with the changes.
> >
>
> Few comments
> -------------------------
> 0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer
> 1.
> + {
> + {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
> + gettext_noop("Sets the maximum memory to be used for logical decoding."),
> + gettext_noop("This much memory can be used by each internal "
> + "reorder buffer before spilling to disk or streaming."),
> + GUC_UNIT_KB
> + },
>
> I think we can remove 'or streaming' from above sentence for now. We
> can add it later with later patch where streaming will be allowed.
Done
>
> 2.
> @@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
> </para>
> </listitem>
> </varlistentry>
> +
> + <varlistentry>
> + <term><literal>work_mem</literal> (<type>integer</type>)</term>
> + <listitem>
> + <para>
> + Limits the amount of memory used to decode changes on the
> + publisher. If not specified, the publisher will use the default
> + specified by <varname>logical_decoding_work_mem</varname>. When
> + needed, additional data are spilled to disk.
> + </para>
> + </listitem>
> + </varlistentry>
>
> It is not clear why we need this parameter at least with this patch?
> I have raised this multiple times [1][2].
I have moved it out as a separate patch (0003) so that if we need that
we need this for the streaming transaction then we can keep this.
>
> bugs_and_review_comments_fix
> 1.
> },
> &logical_decoding_work_mem,
> - -1, -1, MAX_KILOBYTES,
> - check_logical_decoding_work_mem, NULL, NULL
> + 65536, 64, MAX_KILOBYTES,
> + NULL, NULL, NULL
>
> I think the default value should be 1MB similar to
> maintenance_work_mem. The same was true before this change.
default value for maintenance_work_mem is also 64MB. Did you mean min value?
>
> 2. -#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
> maintenance_work_mem
> +i#logical_decoding_work_mem = 64MB # min 64kB
>
> It seems the 'i' is a leftover character in the above change. Also,
> change the default value considering the previous point.
oops, fixed.
>
> 3.
> @@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb,
> ReorderBufferTXN *txn)
>
> /* update the statistics */
> rb->spillCount += 1;
> - rb->spillTxns += txn->serialized ? 1 : 0;
> + rb->spillTxns += txn->serialized ? 0 : 1;
> rb->spillBytes += size;
>
> Why is this change required? Shouldn't we increase the spillTxns
> count only when the txn is serialized?
Already agreed in previous mail so added comments
>
> 0002-Track-statistics-for-spilling
> 1.
> + <row>
> + <entry><structfield>spill_txns</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of transactions spilled to disk after the memory used by
> + logical decoding exceeds <literal>logical_work_mem</literal>. The
> + counter gets incremented both for toplevel transactions and
> + subtransactions.
> + </entry>
> + </row>
>
> The parameter name is wrong here. /logical_work_mem/logical_decoding_work_mem
done
>
> 2.
> + <row>
> + <entry><structfield>spill_txns</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of transactions spilled to disk after the memory used by
> + logical decoding exceeds <literal>logical_work_mem</literal>. The
> + counter gets incremented both for toplevel transactions and
> + subtransactions.
> + </entry>
> + </row>
> + <row>
> + <entry><structfield>spill_count</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Number of times transactions were spilled to disk. Transactions
> + may get spilled repeatedly, and this counter gets incremented on every
> + such invocation.
> + </entry>
> + </row>
> + <row>
> + <entry><structfield>spill_bytes</structfield></entry>
> + <entry><type>integer</type></entry>
> + <entry>Amount of decoded transaction data spilled to disk.
> + </entry>
> + </row>
>
> In all the above cases, the explanation text starts immediately after
> <entry> tag, but the general coding practice is to start from the next
> line, see the explanation of nearby parameters.
It seems it's mixed, for example, you can see
<entry>Timeline number of last write-ahead log location received and
flushed to disk, the initial value of this field being the timeline
number of the first log location used when WAL receiver is started
</entry>
or
<entry>Timeline number of last write-ahead log location received and
flushed to disk, the initial value of this field being the timeline
number of the first log location used when WAL receiver is started
</entry>
>
> It seems these parameters are added in pg-stat-wal-receiver-view in
> the docs, but in code, it is present as part of pg_stat_replication.
> It seems doc needs to be updated. Am, I missing something?
Fixed
>
> 3.
> ReorderBufferSerializeTXN()
> {
> ..
> /* update the statistics */
> rb->spillCount += 1;
> rb->spillTxns += txn->serialized ? 0 : 1;
> rb->spillBytes += size;
>
> Assert(spilled == txn->nentries_mem);
> Assert(dlist_is_empty(&txn->changes));
> txn->nentries_mem = 0;
> txn->serialized = true;
> ..
> }
>
> I am not able to understand the above code. We are setting the
> serialized parameter a few lines after we check it and increment the
> spillTxns count. Can you please explain it?
>
> Also, isn't spillTxns count bit confusing, because in some cases it
> will include subtransactions and other cases (where the largest picked
> transaction is a subtransaction) it won't include it?
>
Already discussed in the last mail.
I have merged bugs_and_review_comments_fix.patch changes to 0001 and 0002.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch (application/octet-stream, 21.3K) download
From e51774752ee74ea51509e469e236f6203e8efc01 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 21 Oct 2019 16:59:17 +0530
Subject: [PATCH 1/3] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to trigger very aggressive streaming. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem may be set either in postgresql.conf,
in which case it serves as the default for all publishers on that
instance, or when creating the subscription, using a work_mem
paramemter in the WITH clause (specifies number of kilobytes).
---
doc/src/sgml/config.sgml | 21 ++
src/backend/replication/logical/reorderbuffer.c | 293 +++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/reorderbuffer.h | 16 ++
5 files changed, 332 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 886632f..291b343 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1716,6 +1716,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 62e5424..8ed8bd8 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is then
+ * serialized to disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when
+ * reaching memory limit. So we just don't count them, because it
+ * might easily trigger a pointless attempt to spill/stream.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2103,6 +2196,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.cmax = cmax;
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
+ change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
dlist_push_tail(&txn->tuplecids, &change->node);
@@ -2230,20 +2324,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2671,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3020,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters).
+ * And we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3249,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3282,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going to modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3438,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 31a5ef0..49ba9cc 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -65,6 +65,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
+#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
@@ -2251,6 +2252,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ 65536, 64, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3..129b3ab 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 64kB
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..4dcef80 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
--
1.8.3.1
0002-Track-statistics-for-spilling.patch (application/octet-stream, 11.2K) download
From e10118b6d211915f040a70b18eb58a4cc32e7051 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 11 Oct 2019 09:07:41 +0530
Subject: [PATCH 2/3] Track statistics for spilling
---
doc/src/sgml/monitoring.sgml | 23 ++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 12 +++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 103 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 828e908..18be607 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1971,6 +1971,29 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<entry><type>timestamp with time zone</type></entry>
<entry>Send time of last reply message received from standby server</entry>
</row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_decoding_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.
+ </entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a47..2ee2a06 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8ed8bd8..1fa0261 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2415,6 +2419,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2473,6 +2478,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillBytes += size;
+
+ /* Don't consider already serialized transaction. */
+ rb->spillTxns += txn->serialized ? 0 : 1;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index b0ebe50..af40d01 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2318,6 +2325,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3219,7 +3229,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3274,6 +3284,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3294,6 +3307,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3375,6 +3391,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3611,3 +3632,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4dcef80..ba7f9f0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 210e9cd..750bdc4 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1951,9 +1951,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0003-Support-logical_decoding_work_mem-set-from-create-su.patch (application/octet-stream, 13.0K) download
From bab5dbc8915d074604614fcf3b8486abb04d3a2d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 21 Oct 2019 17:45:47 +0530
Subject: [PATCH 3/3] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 291b343..6b700c6 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1737,6 +1737,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee283..7e3ba8e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e67a58..d85e831 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +98,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -182,6 +185,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +338,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -341,7 +356,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -419,6 +434,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -682,10 +703,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -710,6 +734,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -721,7 +752,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -759,7 +791,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -796,7 +828,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a..65b3266 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303..14c0ce8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1726,6 +1726,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757..317c5d4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/inval.h"
#include "utils/int8.h"
#include "utils/memutils.h"
@@ -90,11 +91,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -140,6 +142,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +199,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 05:16 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-22 05:16 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; a.kondratov@postgrespro.ru; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have attempted to test the performance of (Stream + Spill) vs
> (Stream + BGW pool) and I can see the similar gain what Alexey had
> shown[1].
>
> In addition to this, I have rebased the latest patchset [2] without
> the two-phase logical decoding patch set.
>
> Test results:
> I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> here is my result
> Stream + Spill
> N time on master(sec) Total xact time (sec)
> 1kk 6 21
> 3kk 18 55
>
> Stream + BGW pool
> N time on master(sec) Total xact time (sec)
> 1kk 6 13
> 3kk 19 35
>
I think the test results for the master are missing. Also, how about
running these tests over a network (means master and subscriber are
not on the same machine)? In general, yours and Alexy's test results
show that there is merit by having workers applying such transactions.
OTOH, as noted above [1], we are also worried about the performance
of Rollbacks if we follow that approach. I am not sure how much we
need to worry about Rollabcks if commits are faster, but can we think
of recording the changes in memory and only write to a file if the
changes are above a certain threshold? I think that might help saving
I/O in many cases. I am not very sure if we do that how much
additional workers can help, but they might still help. I think we
need to do some tests and experiments to figure out what is the best
approach? What do you think?
Tomas, Alexey, do you have any thoughts on this matter? I think it is
important that we figure out the way to proceed in this patch.
[1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 05:31 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-22 05:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: a.kondratov@postgrespro.ru, Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have attempted to test the performance of (Stream + Spill) vs
> > (Stream + BGW pool) and I can see the similar gain what Alexey had
> > shown[1].
> >
> > In addition to this, I have rebased the latest patchset [2] without
> > the two-phase logical decoding patch set.
> >
> > Test results:
> > I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> > here is my result
> > Stream + Spill
> > N time on master(sec) Total xact time (sec)
> > 1kk 6 21
> > 3kk 18 55
> >
> > Stream + BGW pool
> > N time on master(sec) Total xact time (sec)
> > 1kk 6 13
> > 3kk 19 35
> >
>
> I think the test results for the master are missing.
Yeah, That time, I was planning to compare spill vs bgworker.
Also, how about
> running these tests over a network (means master and subscriber are
> not on the same machine)?
Yeah, we should do that that will show the merit of streaming the
in-progress transactions.
In general, yours and Alexy's test results
> show that there is merit by having workers applying such transactions.
> OTOH, as noted above [1], we are also worried about the performance
> of Rollbacks if we follow that approach. I am not sure how much we
> need to worry about Rollabcks if commits are faster, but can we think
> of recording the changes in memory and only write to a file if the
> changes are above a certain threshold? I think that might help saving
> I/O in many cases. I am not very sure if we do that how much
> additional workers can help, but they might still help. I think we
> need to do some tests and experiments to figure out what is the best
> approach? What do you think?
I agree with the point. I think we might need to do some small
changes and test to see what could be the best method to handle the
streamed changes at the subscriber end.
>
> Tomas, Alexey, do you have any thoughts on this matter? I think it is
> important that we figure out the way to proceed in this patch.
>
> [1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 17:12 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-10-22 17:12 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:30:16AM +0530, Dilip Kumar wrote:
>On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Oct 14, 2019 at 3:09 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> > On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
>> > <tomas.vondra@2ndquadrant.com> wrote:
>> > >
>> > >
>> > > Sure, I wasn't really proposing to adding all stats from that patch,
>> > > including those related to streaming. We need to extract just those
>> > > related to spilling. And yes, it needs to be moved right after 0001.
>> > >
>> > I have extracted the spilling related code to a separate patch on top
>> > of 0001. I have also fixed some bugs and review comments and attached
>> > as a separate patch. Later I can merge it to the main patch if you
>> > agree with the changes.
>> >
>>
>> Few comments
>> -------------------------
>> 0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer
>> 1.
>> + {
>> + {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
>> + gettext_noop("Sets the maximum memory to be used for logical decoding."),
>> + gettext_noop("This much memory can be used by each internal "
>> + "reorder buffer before spilling to disk or streaming."),
>> + GUC_UNIT_KB
>> + },
>>
>> I think we can remove 'or streaming' from above sentence for now. We
>> can add it later with later patch where streaming will be allowed.
>Done
>>
>> 2.
>> @@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable
>> class="parameter">subscription_name</replaceabl
>> </para>
>> </listitem>
>> </varlistentry>
>> +
>> + <varlistentry>
>> + <term><literal>work_mem</literal> (<type>integer</type>)</term>
>> + <listitem>
>> + <para>
>> + Limits the amount of memory used to decode changes on the
>> + publisher. If not specified, the publisher will use the default
>> + specified by <varname>logical_decoding_work_mem</varname>. When
>> + needed, additional data are spilled to disk.
>> + </para>
>> + </listitem>
>> + </varlistentry>
>>
>> It is not clear why we need this parameter at least with this patch?
>> I have raised this multiple times [1][2].
>
>I have moved it out as a separate patch (0003) so that if we need that
>we need this for the streaming transaction then we can keep this.
>>
I'm OK with moving it to a separate patch. That being said I think
ability to control memory usage for individual subscriptions is very
useful. Saying "We don't need such parameter" is essentially equivalent
to saying "One size fits all" and I think we know that's not true.
Imagine a system with multiple subscriptions, some of them mostly
replicating OLTP changes, but one or two replicating tables that are
updated in batches. What we'd have is to allow higher limit for the
batch subscriptions, but much lower limit for the OLTP ones (which they
should never hit in practice).
With a single global GUC, you'll either have a high value - risking
OOM when the OLTP subscriptions happen to decode a batch update, or a
low value affecting the batch subscriotions.
It's not strictly necessary (and we already have such limit), so I'm OK
with treating it as an enhancement for the future.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 17:22 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-10-22 17:22 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 11:01:48AM +0530, Dilip Kumar wrote:
>On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> > I have attempted to test the performance of (Stream + Spill) vs
>> > (Stream + BGW pool) and I can see the similar gain what Alexey had
>> > shown[1].
>> >
>> > In addition to this, I have rebased the latest patchset [2] without
>> > the two-phase logical decoding patch set.
>> >
>> > Test results:
>> > I have repeated the same test as Alexy[1] for 1kk and 1kk data and
>> > here is my result
>> > Stream + Spill
>> > N time on master(sec) Total xact time (sec)
>> > 1kk 6 21
>> > 3kk 18 55
>> >
>> > Stream + BGW pool
>> > N time on master(sec) Total xact time (sec)
>> > 1kk 6 13
>> > 3kk 19 35
>> >
>>
>> I think the test results for the master are missing.
>Yeah, That time, I was planning to compare spill vs bgworker.
> Also, how about
>> running these tests over a network (means master and subscriber are
>> not on the same machine)?
>
>Yeah, we should do that that will show the merit of streaming the
>in-progress transactions.
>
Which I agree it's an interesting feature, I think we need to stop
adding more stuff to this patch series - it's already complex enough, so
making it even more (unnecessary) stuff is a distraction and will make
it harder to get anything committed. Typical "scope creep".
I think the current behavior (spill to file) is sufficient for v0 and
can be improved later - that's fine. I don't think we need to bother
with comparisons to master very much, because while it might be a bit
slower in some cases, you can always disable streaming (so if there's a
regression for your workload, you can undo that).
> In general, yours and Alexy's test results
>> show that there is merit by having workers applying such transactions.
>> OTOH, as noted above [1], we are also worried about the performance
>> of Rollbacks if we follow that approach. I am not sure how much we
>> need to worry about Rollabcks if commits are faster, but can we think
>> of recording the changes in memory and only write to a file if the
>> changes are above a certain threshold? I think that might help saving
>> I/O in many cases. I am not very sure if we do that how much
>> additional workers can help, but they might still help. I think we
>> need to do some tests and experiments to figure out what is the best
>> approach? What do you think?
>I agree with the point. I think we might need to do some small
>changes and test to see what could be the best method to handle the
>streamed changes at the subscriber end.
>
>>
>> Tomas, Alexey, do you have any thoughts on this matter? I think it is
>> important that we figure out the way to proceed in this patch.
>>
>> [1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>>
>
I think the patch should do the simplest thing possible, i.e. what it
does today. Otherwise we'll never get it committed.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-22 19:02 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-10-22 19:02 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 22.10.2019 20:22, Tomas Vondra wrote:
> On Tue, Oct 22, 2019 at 11:01:48AM +0530, Dilip Kumar wrote:
>> On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila
>> <amit.kapila16@gmail.com> wrote:
>> In general, yours and Alexy's test results
>>> show that there is merit by having workers applying such transactions.
>>> OTOH, as noted above [1], we are also worried about the performance
>>> of Rollbacks if we follow that approach. I am not sure how much we
>>> need to worry about Rollabcks if commits are faster, but can we think
>>> of recording the changes in memory and only write to a file if the
>>> changes are above a certain threshold? I think that might help saving
>>> I/O in many cases. I am not very sure if we do that how much
>>> additional workers can help, but they might still help. I think we
>>> need to do some tests and experiments to figure out what is the best
>>> approach? What do you think?
>> I agree with the point. I think we might need to do some small
>> changes and test to see what could be the best method to handle the
>> streamed changes at the subscriber end.
>>
>>>
>>> Tomas, Alexey, do you have any thoughts on this matter? I think it is
>>> important that we figure out the way to proceed in this patch.
>>>
>>> [1] -
>>> https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>>>
>>
>
> I think the patch should do the simplest thing possible, i.e. what it
> does today. Otherwise we'll never get it committed.
>
I have to agree with Tomas, that keeping things as simple as possible
should be a main priority right now. Otherwise, the entire patch set
will pass next release cycle without being committed at least partially.
In the same time, it resolves important problem from my perspective. It
moves I/O overhead from primary to replica using large transactions
streaming, which is a nice to have feature I guess.
Later it would be possible to replace logical apply worker with
bgworkers pool in a separated patch, if we decide that it is a viable
solution. Anyway, regarding the Amit's questions:
- I doubt that maintaining a separate buffer on the apply side before
spilling to disk would help enough. We already have ReorderBuffer with
logical_work_mem limit, and if we exceeded that limit on the sender
side, then most probably we exceed it on the applier side as well,
excepting the case when this new buffer size will be significantly
higher then logical_work_mem to keep multiple open xacts.
- I still think that we should optimize database for commits, not
rollbacks. BGworkers pool is dramatically slower for rollbacks-only
load, though being at least twice as faster for commits-only. I do not
know how it will perform with real life load, but this drawback may be
inappropriate for such a general purpose database like Postgres.
- Tomas' implementation of streaming with spilling does not have this
bias between commits/aborts. However, it has a noticeable performance
drop (~x5 slower compared with master [1]) for large transaction
consisting of many small rows. Although it is not of an order of
magnitude slower.
Another thing is it that about a year ago I have found some problems
with MVCC/visibility and fixed them somehow [1]. If I get it correctly
Tomas adapted some of those fixes into his patch set, but I think that
this part should be reviewed carefully again. I would be glad to check
it, but now I am a little bit confused with all the patch set variants
in the thread. Which is the last one? Is it still dependent on 2pc decoding?
[1]
https://www.postgresql.org/message-id/flat/40c38758-04b5-74f4-c963-cf300f9e5dff%40postgrespro.ru#98d...
Thanks for moving this patch forward!
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-23 09:13 Amit Kapila <amit.kapila16@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-10-23 09:13 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:42 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Tue, Oct 22, 2019 at 10:30:16AM +0530, Dilip Kumar wrote:
> >
> >I have moved it out as a separate patch (0003) so that if we need that
> >we need this for the streaming transaction then we can keep this.
> >>
>
> I'm OK with moving it to a separate patch. That being said I think
> ability to control memory usage for individual subscriptions is very
> useful. Saying "We don't need such parameter" is essentially equivalent
> to saying "One size fits all" and I think we know that's not true.
>
> Imagine a system with multiple subscriptions, some of them mostly
> replicating OLTP changes, but one or two replicating tables that are
> updated in batches. What we'd have is to allow higher limit for the
> batch subscriptions, but much lower limit for the OLTP ones (which they
> should never hit in practice).
>
This point is not clear to me. The changes are recorded in
ReorderBuffer which doesn't have any filtering aka it will have all
the changes irrespective of the subscriber. How will it make a
difference to have different limits?
> With a single global GUC, you'll either have a high value - risking
> OOM when the OLTP subscriptions happen to decode a batch update, or a
> low value affecting the batch subscriotions.
>
> It's not strictly necessary (and we already have such limit), so I'm OK
> with treating it as an enhancement for the future.
>
I am fine too if its usage is clear. I might be missing something here.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-23 10:50 Amit Kapila <amit.kapila16@gmail.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-10-23 10:50 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 23, 2019 at 12:32 AM Alexey Kondratov
<a.kondratov@postgrespro.ru> wrote:
>
> On 22.10.2019 20:22, Tomas Vondra wrote:
> >
> > I think the patch should do the simplest thing possible, i.e. what it
> > does today. Otherwise we'll never get it committed.
> >
>
> I have to agree with Tomas, that keeping things as simple as possible
> should be a main priority right now. Otherwise, the entire patch set
> will pass next release cycle without being committed at least partially.
> In the same time, it resolves important problem from my perspective. It
> moves I/O overhead from primary to replica using large transactions
> streaming, which is a nice to have feature I guess.
>
> Later it would be possible to replace logical apply worker with
> bgworkers pool in a separated patch, if we decide that it is a viable
> solution. Anyway, regarding the Amit's questions:
>
> - I doubt that maintaining a separate buffer on the apply side before
> spilling to disk would help enough. We already have ReorderBuffer with
> logical_work_mem limit, and if we exceeded that limit on the sender
> side, then most probably we exceed it on the applier side as well,
>
I think on the sender side, the limit is for un-filtered changes
(which means on the ReorderBuffer which has all the changes) whereas,
on the receiver side, we will only have the requested changes which
can make a difference?
> excepting the case when this new buffer size will be significantly
> higher then logical_work_mem to keep multiple open xacts.
>
I am not sure but I think we can have different controlling parameters
on the subscriber-side.
> - I still think that we should optimize database for commits, not
> rollbacks. BGworkers pool is dramatically slower for rollbacks-only
> load, though being at least twice as faster for commits-only. I do not
> know how it will perform with real life load, but this drawback may be
> inappropriate for such a general purpose database like Postgres.
>
> - Tomas' implementation of streaming with spilling does not have this
> bias between commits/aborts. However, it has a noticeable performance
> drop (~x5 slower compared with master [1]) for large transaction
> consisting of many small rows. Although it is not of an order of
> magnitude slower.
>
Did you ever identify the reason why it was slower in that case? I
can see the numbers shared by you and Dilip which shows that the
BGWorker pool is a really good idea and will work great for
commit-mostly workload whereas the numbers without that are not very
encouraging, maybe we have not benchmarked enough. This is the reason
I am trying to see if we can do something to get the benefits similar
to what is shown by your idea.
I am not against doing something simple for the first version and then
enhance it later, but it won't be good if we commit it with regression
in some typical cases and depend on the user to use it when it seems
favorable to its case. Also, sometimes it becomes difficult to
generate enthusiasm to enhance the feature once the main patch is
committed. I am not telling that always happens or will happen in
this case. It is better if we put some energy and get things as good
as possible in the first go itself. I am as much interested as you,
Tomas or others are, otherwise, I wouldn't have spent a lot of time on
this to disentangle it from 2PC patch which seems to get stalled due
to lack of interest.
> Another thing is it that about a year ago I have found some problems
> with MVCC/visibility and fixed them somehow [1]. If I get it correctly
> Tomas adapted some of those fixes into his patch set, but I think that
> this part should be reviewed carefully again.
>
Agreed, I have read your emails and could see that you have done very
good work on this project along with Tomas. But unfortunately, it
didn't get committed. At this stage, we are working on just the first
part of the patch which is to allow the data to spill once it crosses
the logical_decoding_work_mem on the master side. I think we need
more problems to discuss and solve once that is done.
> I would be glad to check
> it, but now I am a little bit confused with all the patch set variants
> in the thread. Which is the last one? Is it still dependent on 2pc decoding?
>
I think the latest patches posted by Dilip are not dependent on
logical decoding, but I haven't studied them yet. You can find those
at [1][2]. As per discussion in this thread, we are also trying to
see if we can make some part of the patch-series committed first, the
latest patches corresponding to which are posted at [3].
[1] - https://www.postgresql.org/message-id/CAFiTN-vHoksqvV4BZ0479NhugGe4QHq_ezngNdDd-YRQ_2cwug%40mail.gma...
[2] - https://www.postgresql.org/message-id/CAFiTN-vT%2B42xRbkw%3DhBnp44XkAyZaKZVA5hcvAMsYth3rk7vhg%40mail...
[3] - https://www.postgresql.org/message-id/CAFiTN-vkFB0RBEjVkLWhdgTYShSrSu3kCYObMghgXEwKA1FXRA%40mail.gma...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-24 13:37 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-24 13:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have merged bugs_and_review_comments_fix.patch changes to 0001 and 0002.
>
I was wondering whether we have checked the code coverage after this
patch? Previously, the existing tests seem to be covering most parts
of the function ReorderBufferSerializeTXN [1]. After this patch, the
timing to call ReorderBufferSerializeTXN will change, so that might
impact the testing of the same. If it is already covered, then I
would like to either add a new test or extend existing test with the
help of new spill counters. If it is not getting covered, then we
need to think of extending the existing test or write a new test to
cover the function ReorderBufferSerializeTXN.
[1] - https://coverage.postgresql.org/src/backend/replication/logical/reorderbuffer.c.gcov.html
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-30 04:08 vignesh C <vignesh21@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 2 replies; 536+ messages in thread
From: vignesh C @ 2019-10-30 04:08 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> I think the patch should do the simplest thing possible, i.e. what it
> does today. Otherwise we'll never get it committed.
>
I found a couple of crashes while reviewing and testing flushing of
open transaction data:
Issue 1:
#0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
#1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
#2 0x0000000000ec5390 in ExceptionalCondition
(conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
"FailedAssertion",
fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
lineNumber=458) at assert.c:54
#3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
off=64) at ../../../../src/include/lib/ilist.h:458
#4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
oldestRunningXid=3834) at reorderbuffer.c:1966
#5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
buf=0x7ffcbc26dc50) at decode.c:332
#6 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x19af990,
record=0x19afc50) at decode.c:121
#7 0x0000000000b7109e in XLogSendLogical () at walsender.c:2845
#8 0x0000000000b6f5e4 in WalSndLoop (send_data=0xb70f77
<XLogSendLogical>) at walsender.c:2199
#9 0x0000000000b6c7e1 in StartLogicalReplication (cmd=0x1983168) at
walsender.c:1128
#10 0x0000000000b6da6f in exec_replication_command
(cmd_string=0x18f70a0 "START_REPLICATION SLOT \"sub1\" LOGICAL 0/0
(proto_version '1', publication_names '\"pub1\"')")
at walsender.c:1545
Issue 2:
#0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
#1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
#2 0x0000000000ec4e1d in ExceptionalCondition
(conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
errorType=0x10ea284 "FailedAssertion",
fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
#3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
txn=0x2bafb08) at reorderbuffer.c:3052
#4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
txn=0x2bafb08) at reorderbuffer.c:1318
#5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
txn=0x2b9d778) at reorderbuffer.c:1257
#6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
oldestRunningXid=3835) at reorderbuffer.c:1973
#7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
buf=0x7ffcbc74cc00) at decode.c:332
#8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
record=0x2b67990) at decode.c:121
#9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
These failures come randomly.
I'm not able to reproduce this issue with simple test case.
I have attached the test case which I used to test.
I will further try to find a scenario which could reproduce consistently.
Posting it so that it can help someone in identifying the problem
parallelly through code review by experts.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
Attachments:
mix_data_test.c (text/x-c-code, 7.1K) download
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>
#define LARGEDATATHREADS 32
#define SMALLDATATHREADS 50
void do_exit(PGconn *conn, PGresult *res)
{
fprintf(stderr, "%s\n", PQerrorMessage(conn));
PQclear(res);
PQfinish(conn);
exit(1);
}
void initsubscriber()
{
PGconn *conn = PQconnectdb("user=user1 dbname=testdb port=5433");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
PGresult *res = PQexec(conn, "DROP TABLE IF EXISTS perftest_smalldata");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest_smalldata(c1 varchar(10), c2 varchar(10), c3 varchar(10), c4 varchar(10), c5 varchar(10))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "DROP TABLE IF EXISTS perftest");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest(c1 varchar(100), c2 varchar(100), c3 varchar(100), c4 varchar(100), c5 varchar(100))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "create subscription sub1 connection 'host=127.0.0.1 port=5432 dbname=testdb user=user1 password=123' publication pub1");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
PQfinish(conn);
}
void initpublisher()
{
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
PGresult *res = PQexec(conn, "DROP TABLE IF EXISTS perftest_smalldata");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest_smalldata(c1 varchar(10), c2 varchar(10), c3 varchar(10), c4 varchar(10), c5 varchar(10))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "DROP TABLE IF EXISTS perftest");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest(c1 varchar(100), c2 varchar(100), c3 varchar(100), c4 varchar(100), c5 varchar(100))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE PUBLICATION pub1 FOR TABLE perftest, perftest_smalldata WITH (publish='insert,update,delete')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
PQfinish(conn);
}
void* smalldatathreadfunc(void *recordcount)
{
unsigned int reccount = *((unsigned int *) recordcount);
int count;
PGresult *res;
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
res = PQexec(conn, "BEGIN");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
/* Load data */
int subtxncount = reccount/64;
int savepointcount = 0;
for (count = 0; count < reccount; count++)
{
res = PQexec(conn, "insert into perftest_smalldata values('asafadfafa', 'asafadfafa', 'asafadfafa', 'asafadfafa', 'asafadfafa')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
if (count % subtxncount == 0)
{
char savepointstr[128] = {0};
savepointcount++;
sprintf(savepointstr, "savepoint sp%d", savepointcount);
res = PQexec(conn, savepointstr);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
}
}
printf("%d subtransaction created\n", savepointcount);
printf("Keeping txn open for 60 seconds\n");
sleep(60);
res = PQexec(conn, "SELECT * FROM perftest_smalldata");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("No data retrieved\n");
PQclear(res);
PQfinish(conn);
exit(1);
}
int rows = PQntuples(res);
printf("Record count = %d\n", rows);
PQclear(res);
PQfinish(conn);
return 0;
}
void* largedatathreadfunc(void *recordcount)
{
unsigned int reccount = *((unsigned int *) recordcount);
int count;
PGresult *res;
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
res = PQexec(conn, "BEGIN");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
/* Load data */
for (count = 0; count < reccount; count++)
{
res = PQexec(conn, "insert into perftest values(\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
}
printf("Keeping txn open for 60 seconds\n");
sleep(60);
res = PQexec(conn, "SELECT * FROM perftest");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("No data retrieved\n");
PQclear(res);
PQfinish(conn);
exit(1);
}
int rows = PQntuples(res);
printf("Record count = %d\n", rows);
PQclear(res);
PQfinish(conn);
return 0;
}
int main()
{
pthread_t largedata_thread_id[LARGEDATATHREADS];
pthread_t smalldata_thread_id[SMALLDATATHREADS];
int i;
int recordcount;
initpublisher();
initsubscriber();
for(i=0; i < LARGEDATATHREADS; i++)
{
recordcount = 2500;
pthread_create( &largedata_thread_id[i], NULL, largedatathreadfunc, &recordcount);
}
for(i=0; i < SMALLDATATHREADS; i++)
{
recordcount = 6000;
pthread_create( &smalldata_thread_id[i], NULL, smalldatathreadfunc, &recordcount);
}
for(i=0; i < LARGEDATATHREADS; i++)
{
pthread_join( largedata_thread_id[i], NULL);
}
for(i=0; i < SMALLDATATHREADS; i++)
{
pthread_join( smalldata_thread_id[i], NULL);
}
}
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-10-30 07:48 Dilip Kumar <dilipbalaut@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-30 07:48 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
>
I have noticed one more problem in the logic of setting the logical
decoding work mem from the create subscription command. Suppose in
subscription command we don't give the work mem then it sends the
garbage value to the walsender and the walsender overwrite its value
with the garbage value. After investigating a bit I have found the
reason for the same.
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
I think the problem is we are unconditionally sending the work_mem as
part of the CREATE REPLICATION SLOT, without checking whether it's
valid or not.
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
Another problem is that there is no handling if the subform->subworkmem is NULL.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-04 09:13 Kuntal Ghosh <kuntalghosh.2007@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-04 09:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Hello hackers,
I've done some performance testing of this feature. Following is my
test case (taken from an earlier thread):
postgres=# CREATE TABLE large_test (num1 bigint, num2 double
precision, num3 double precision);
postgres=# \timing on
postgres=# EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1,
num2, num3) SELECT round(random()*10), random(), random()*142 FROM
generate_series(1, 1000000) s(i);
I've kept the publisher and subscriber in two different system.
HEAD:
With 1000000 tuples,
Execution Time: 2576.821 ms, Time: 9.632.158 ms (00:09.632), Spill count: 245
With 10000000 tuples (10 times more),
Execution Time: 30359.509 ms, Time: 95261.024 ms (01:35.261), Spill count: 2442
With the memory accounting patch, following are the performance results:
With 100000 tuples,
logical_decoding_work_mem=64kB, Execution Time: 2414.371 ms, Time:
9648.223 ms (00:09.648), Spill count: 2315
logical_decoding_work_mem=64MB, Execution Time: 2477.830 ms, Time:
9895.161 ms (00:09.895), Spill count 3
With 1000000 tuples (10 times more),
logical_decoding_work_mem=64kB, Execution Time: 38259.227 ms, Time:
105761.978 ms (01:45.762), Spill count: 23149
logical_decoding_work_mem=64MB, Execution Time: 24624.639 ms, Time:
89985.342 ms (01:29.985), Spill count: 23
With logical decoding of in-progress transactions patch and with
streaming on, following are the performance results:
With 100000 tuples,
logical_decoding_work_mem=64kB, Execution Time: 2674.034 ms, Time:
20779.601 ms (00:20.780)
logical_decoding_work_mem=64MB, Execution Time: 2062.404 ms, Time:
9559.953 ms (00:09.560)
With 1000000 tuples (10 times more),
logical_decoding_work_mem=64kB, Execution Time: 26949.588 ms, Time:
196261.892 ms (03:16.262)
logical_decoding_work_mem=64MB, Execution Time: 27084.403 ms, Time:
90079.286 ms (01:30.079)
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-04 10:02 Dilip Kumar <dilipbalaut@gmail.com>
parent: Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-04 10:02 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 2:43 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> Hello hackers,
>
> I've done some performance testing of this feature. Following is my
> test case (taken from an earlier thread):
>
> postgres=# CREATE TABLE large_test (num1 bigint, num2 double
> precision, num3 double precision);
> postgres=# \timing on
> postgres=# EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1,
> num2, num3) SELECT round(random()*10), random(), random()*142 FROM
> generate_series(1, 1000000) s(i);
>
> I've kept the publisher and subscriber in two different system.
>
> HEAD:
> With 1000000 tuples,
> Execution Time: 2576.821 ms, Time: 9.632.158 ms (00:09.632), Spill count: 245
> With 10000000 tuples (10 times more),
> Execution Time: 30359.509 ms, Time: 95261.024 ms (01:35.261), Spill count: 2442
>
> With the memory accounting patch, following are the performance results:
> With 100000 tuples,
> logical_decoding_work_mem=64kB, Execution Time: 2414.371 ms, Time:
> 9648.223 ms (00:09.648), Spill count: 2315
> logical_decoding_work_mem=64MB, Execution Time: 2477.830 ms, Time:
> 9895.161 ms (00:09.895), Spill count 3
> With 1000000 tuples (10 times more),
> logical_decoding_work_mem=64kB, Execution Time: 38259.227 ms, Time:
> 105761.978 ms (01:45.762), Spill count: 23149
> logical_decoding_work_mem=64MB, Execution Time: 24624.639 ms, Time:
> 89985.342 ms (01:29.985), Spill count: 23
>
> With logical decoding of in-progress transactions patch and with
> streaming on, following are the performance results:
> With 100000 tuples,
> logical_decoding_work_mem=64kB, Execution Time: 2674.034 ms, Time:
> 20779.601 ms (00:20.780)
> logical_decoding_work_mem=64MB, Execution Time: 2062.404 ms, Time:
> 9559.953 ms (00:09.560)
> With 1000000 tuples (10 times more),
> logical_decoding_work_mem=64kB, Execution Time: 26949.588 ms, Time:
> 196261.892 ms (03:16.262)
> logical_decoding_work_mem=64MB, Execution Time: 27084.403 ms, Time:
> 90079.286 ms (01:30.079)
So your result shows that with "streaming on", performance is
degrading? By any chance did you try to see where is the bottleneck?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-04 10:05 Kuntal Ghosh <kuntalghosh.2007@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-04 10:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> So your result shows that with "streaming on", performance is
> degrading? By any chance did you try to see where is the bottleneck?
>
Right. But, as we increase the logical_decoding_work_mem, the
performance improves. I've not analyzed the bottleneck yet. I'm
looking into the same.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-04 10:16 vignesh C <vignesh21@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: vignesh C @ 2019-11-04 10:16 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 24, 2019 at 7:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Oct 22, 2019 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have merged bugs_and_review_comments_fix.patch changes to 0001 and 0002.
> >
>
> I was wondering whether we have checked the code coverage after this
> patch? Previously, the existing tests seem to be covering most parts
> of the function ReorderBufferSerializeTXN [1]. After this patch, the
> timing to call ReorderBufferSerializeTXN will change, so that might
> impact the testing of the same. If it is already covered, then I
> would like to either add a new test or extend existing test with the
> help of new spill counters. If it is not getting covered, then we
> need to think of extending the existing test or write a new test to
> cover the function ReorderBufferSerializeTXN.
>
I have run the tests with coverage and found that
ReorderBufferSerializeTXN is not being hit.
The reason it is not being hit is because of the following check in
ReorderBufferCheckMemoryLimit:
/* bail out if we haven't exceeded the memory limit */
if (rb->size < logical_decoding_work_mem * 1024L)
return;
Previously the tests from contrib/test_decoding could hit
ReorderBufferSerializeTXN function.
I'm checking if we can modify the test or add new test to hit
ReorderBufferSerializeTXN function.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-04 11:51 Amit Kapila <amit.kapila16@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-11-04 11:51 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> > I think the patch should do the simplest thing possible, i.e. what it
> > does today. Otherwise we'll never get it committed.
> >
> I found a couple of crashes while reviewing and testing flushing of
> open transaction data:
>
Thanks for doing these tests. However, I don't think these issues are
anyway related to this patch. It seems to be base code issues
manifested by this patch. See my analysis below.
> Issue 1:
> #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> #2 0x0000000000ec5390 in ExceptionalCondition
> (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> "FailedAssertion",
> fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> lineNumber=458) at assert.c:54
> #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> off=64) at ../../../../src/include/lib/ilist.h:458
> #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> oldestRunningXid=3834) at reorderbuffer.c:1966
> #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> buf=0x7ffcbc26dc50) at decode.c:332
>
This seems to be the problem of base code where we abort immediately
after serializing the changes because in that case, the changes list
will be empty. I think you can try to reproduce it via the debugger
or by hacking the code such that it serializes after every change and
then if you abort after one change, it should hit this problem.
>
> Issue 2:
> #0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
> #1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
> #2 0x0000000000ec4e1d in ExceptionalCondition
> (conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
> errorType=0x10ea284 "FailedAssertion",
> fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
> #3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
> txn=0x2bafb08) at reorderbuffer.c:3052
> #4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
> txn=0x2bafb08) at reorderbuffer.c:1318
> #5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
> txn=0x2b9d778) at reorderbuffer.c:1257
> #6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
> oldestRunningXid=3835) at reorderbuffer.c:1973
>
This seems to be again the problem with base code as we don't update
the final_lsn for subtransactions during ReorderBufferAbortOld. This
can also be reproduced with some hacking in code or via debugger in a
similar way as explained for the previous problem but with a
difference that there must be subtransaction involved in this case.
> #7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
> buf=0x7ffcbc74cc00) at decode.c:332
> #8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
> record=0x2b67990) at decode.c:121
> #9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
>
> These failures come randomly.
> I'm not able to reproduce this issue with simple test case.
Yeah, it appears to be difficult to reproduce unless you hack the code
to serialize every change or use debugger to forcefully flush the
changes every time.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-05 05:38 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-11-05 05:38 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > > I think the patch should do the simplest thing possible, i.e. what it
> > > does today. Otherwise we'll never get it committed.
> > >
> > I found a couple of crashes while reviewing and testing flushing of
> > open transaction data:
> >
>
> Thanks for doing these tests. However, I don't think these issues are
> anyway related to this patch. It seems to be base code issues
> manifested by this patch. See my analysis below.
>
> > Issue 1:
> > #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> > #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec5390 in ExceptionalCondition
> > (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> > "FailedAssertion",
> > fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> > lineNumber=458) at assert.c:54
> > #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> > off=64) at ../../../../src/include/lib/ilist.h:458
> > #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> > oldestRunningXid=3834) at reorderbuffer.c:1966
> > #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> > buf=0x7ffcbc26dc50) at decode.c:332
> >
>
> This seems to be the problem of base code where we abort immediately
> after serializing the changes because in that case, the changes list
> will be empty. I think you can try to reproduce it via the debugger
> or by hacking the code such that it serializes after every change and
> then if you abort after one change, it should hit this problem.
>
I think you might need to kill the server after all changes are
serialized otherwise normal abort will hit the ReorderBufferAbort and
that will remove your ReorderBufferTXN entry and you will never hit
this case.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-06 06:03 vignesh C <vignesh21@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: vignesh C @ 2019-11-06 06:03 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 3:46 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, Oct 24, 2019 at 7:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have merged bugs_and_review_comments_fix.patch changes to 0001 and 0002.
> > >
> >
> > I was wondering whether we have checked the code coverage after this
> > patch? Previously, the existing tests seem to be covering most parts
> > of the function ReorderBufferSerializeTXN [1]. After this patch, the
> > timing to call ReorderBufferSerializeTXN will change, so that might
> > impact the testing of the same. If it is already covered, then I
> > would like to either add a new test or extend existing test with the
> > help of new spill counters. If it is not getting covered, then we
> > need to think of extending the existing test or write a new test to
> > cover the function ReorderBufferSerializeTXN.
> >
> I have run the tests with coverage and found that
> ReorderBufferSerializeTXN is not being hit.
> The reason it is not being hit is because of the following check in
> ReorderBufferCheckMemoryLimit:
> /* bail out if we haven't exceeded the memory limit */
> if (rb->size < logical_decoding_work_mem * 1024L)
> return;
> Previously the tests from contrib/test_decoding could hit
> ReorderBufferSerializeTXN function.
> I'm checking if we can modify the test or add new test to hit
> ReorderBufferSerializeTXN function.
I have made one change to the configuration file in
contrib/test_decoding directory, with that the coverage seems to be
fine. I have seen that the coverage is almost like the code before
applying the patch. I have attached the test change and the coverage
report for reference. Coverage report includes the core logical work
memory files for base code and by applying
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer and
0002-Track-statistics-for-spilling patches.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-configuration-for-test.patch (text/x-patch, 747B) download
From b86ddac054ad29ac8a48e1e49432f338ef8f947b Mon Sep 17 00:00:00 2001
From: vignesh <vignesh@localhost.localdomain>
Date: Wed, 6 Nov 2019 09:50:48 +0530
Subject: [PATCH] Add logical_decoding_work_mem configuration for test
configuration.
Added logical_decoding_work_mem to test configuration, setting it
to minimum value so that all test paths are covered.
---
contrib/test_decoding/logical.conf | 1 +
1 file changed, 1 insertion(+)
diff --git a/contrib/test_decoding/logical.conf b/contrib/test_decoding/logical.conf
index 367f706..07c4d3d 100644
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,2 +1,3 @@
wal_level = logical
max_replication_slots = 4
+logical_decoding_work_mem = 64kB
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-06 11:03 vignesh C <vignesh21@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: vignesh C @ 2019-11-06 11:03 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > > I think the patch should do the simplest thing possible, i.e. what it
> > > does today. Otherwise we'll never get it committed.
> > >
> > I found a couple of crashes while reviewing and testing flushing of
> > open transaction data:
> >
>
> Thanks for doing these tests. However, I don't think these issues are
> anyway related to this patch. It seems to be base code issues
> manifested by this patch. See my analysis below.
>
> > Issue 1:
> > #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> > #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec5390 in ExceptionalCondition
> > (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> > "FailedAssertion",
> > fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> > lineNumber=458) at assert.c:54
> > #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> > off=64) at ../../../../src/include/lib/ilist.h:458
> > #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> > oldestRunningXid=3834) at reorderbuffer.c:1966
> > #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> > buf=0x7ffcbc26dc50) at decode.c:332
> >
>
> This seems to be the problem of base code where we abort immediately
> after serializing the changes because in that case, the changes list
> will be empty. I think you can try to reproduce it via the debugger
> or by hacking the code such that it serializes after every change and
> then if you abort after one change, it should hit this problem.
>
> >
> > Issue 2:
> > #0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
> > #1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec4e1d in ExceptionalCondition
> > (conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
> > errorType=0x10ea284 "FailedAssertion",
> > fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
> > #3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
> > txn=0x2bafb08) at reorderbuffer.c:3052
> > #4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
> > txn=0x2bafb08) at reorderbuffer.c:1318
> > #5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
> > txn=0x2b9d778) at reorderbuffer.c:1257
> > #6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
> > oldestRunningXid=3835) at reorderbuffer.c:1973
> >
>
> This seems to be again the problem with base code as we don't update
> the final_lsn for subtransactions during ReorderBufferAbortOld. This
> can also be reproduced with some hacking in code or via debugger in a
> similar way as explained for the previous problem but with a
> difference that there must be subtransaction involved in this case.
>
> > #7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
> > buf=0x7ffcbc74cc00) at decode.c:332
> > #8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
> > record=0x2b67990) at decode.c:121
> > #9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
> >
> > These failures come randomly.
> > I'm not able to reproduce this issue with simple test case.
>
> Yeah, it appears to be difficult to reproduce unless you hack the code
> to serialize every change or use debugger to forcefully flush the
> changes every time.
>
Thanks Amit for your analysis, I was able to reproduce the above issue
consistently by making some code changes and with help of debugger. I
did one change so that it flushes every time instead of flushing after
the buffer size exceeds the logical_decoding_work_mem, attached one of
the transactions and called abort. When the server restarts after
abort, this problem occurs consistently. I could reproduce the issue
with base code also. It seems like this issue is not an issue of
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer patch and
exists from base code. I will post the issue in hackers with details.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-07 09:49 Amit Kapila <amit.kapila16@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-07 09:49 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 6, 2019 at 11:33 AM vignesh C <vignesh21@gmail.com> wrote:
>
> I have made one change to the configuration file in
> contrib/test_decoding directory, with that the coverage seems to be
> fine. I have seen that the coverage is almost like the code before
> applying the patch. I have attached the test change and the coverage
> report for reference. Coverage report includes the core logical work
> memory files for base code and by applying
> 0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer and
> 0002-Track-statistics-for-spilling patches.
>
Thanks, I have incorporated your test changes and modified the two
patches. Please see attached.
Changes:
---------------
1. In guc.c, we should include reorderbuffer.h, not logical.h as we
define logical_decoding_work_mem in earlier.
2.
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is then
+ * serialized to disk.
In the above comments, removed 'toplevel' as we track memory usage for
both toplevel and subtransactions.
3. There were still a few mentions of streaming which I have removed.
4. In the docs, the type for stats spill_* was integer whereas it
should be bigint.
5.
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
Changed the above elog to DEBUG1 as otherwise it was getting printed
very frequently. I think we can make it DEBUG2 if we want.
6. There was an extra space in rules.out due to which test was
failing. I have fixed it.
What do you think?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch (application/octet-stream, 21.6K) download
From 4a00f6da1d7254151bf6eb27f55ba238471ac152 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 21 Oct 2019 16:59:17 +0530
Subject: [PATCH 1/2] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to trigger very aggressive streaming. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem may be set either in postgresql.conf,
in which case it serves as the default for all publishers on that
instance, or when creating the subscription, using a work_mem
paramemter in the WITH clause (specifies number of kilobytes).
---
contrib/test_decoding/logical.conf | 1 +
doc/src/sgml/config.sgml | 21 ++
src/backend/replication/logical/reorderbuffer.c | 293 +++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/reorderbuffer.h | 16 ++
6 files changed, 333 insertions(+), 12 deletions(-)
diff --git a/contrib/test_decoding/logical.conf b/contrib/test_decoding/logical.conf
index 367f706..07c4d3d 100644
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,2 +1,3 @@
wal_level = logical
max_replication_slots = 4
+logical_decoding_work_mem = 64kB
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 46bc31d..04ef505 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1732,6 +1732,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are written to local disk. This
+ limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 62e5424..3576d4e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each transaction. When the total amount of used memory exceeds the
+ * limit, the transaction consuming the most memory is then serialized to
+ * disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when
+ * reaching memory limit. So we just don't count them, because it
+ * might easily trigger a pointless attempt to spill.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2103,6 +2196,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.cmax = cmax;
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
+ change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
dlist_push_tail(&txn->tuplecids, &change->node);
@@ -2230,20 +2324,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2671,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3020,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters).
+ * And we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3249,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3282,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going to modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3438,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index e84c8cc..d899995 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -66,6 +66,7 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "replication/walreceiver.h"
@@ -2253,6 +2254,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ 65536, 64, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index be02a76..46a06ff 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 64kB
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..4dcef80 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
--
1.8.3.1
0002-Track-statistics-for-spilling.patch (application/octet-stream, 11.1K) download
From d2b8ae2ab086cd91cccea79e6c5decaaa66fcfdf Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 11 Oct 2019 09:07:41 +0530
Subject: [PATCH 2/2] Track statistics for spilling
---
doc/src/sgml/monitoring.sgml | 20 ++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 12 +++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d18b271..eea65ee 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1971,6 +1971,26 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<entry><type>timestamp with time zone</type></entry>
<entry>Send time of last reply message received from standby server</entry>
</row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_decoding_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.</entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a47..2ee2a06 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3576d4e..a0c66e6 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2415,6 +2419,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2473,6 +2478,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillBytes += size;
+
+ /* Don't consider already serialized transaction. */
+ rb->spillTxns += txn->serialized ? 0 : 1;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 7f56715..d7ef634 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2318,6 +2325,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3219,7 +3229,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3274,6 +3284,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3294,6 +3307,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3375,6 +3391,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3611,3 +3632,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(DEBUG1, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4dcef80..ba7f9f0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 210e9cd..4f1998e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1951,9 +1951,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-07 10:19 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-07 10:19 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 7, 2019 at 3:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Nov 6, 2019 at 11:33 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > I have made one change to the configuration file in
> > contrib/test_decoding directory, with that the coverage seems to be
> > fine. I have seen that the coverage is almost like the code before
> > applying the patch. I have attached the test change and the coverage
> > report for reference. Coverage report includes the core logical work
> > memory files for base code and by applying
> > 0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer and
> > 0002-Track-statistics-for-spilling patches.
> >
>
> Thanks, I have incorporated your test changes and modified the two
> patches. Please see attached.
>
> Changes:
> ---------------
> 1. In guc.c, we should include reorderbuffer.h, not logical.h as we
> define logical_decoding_work_mem in earlier.
Yeah Right.
>
> 2.
> + * To limit the amount of memory used by decoded changes, we track memory
> + * used at the reorder buffer level (i.e. total amount of memory), and for
> + * each toplevel transaction. When the total amount of used memory exceeds
> + * the limit, the toplevel transaction consuming the most memory is then
> + * serialized to disk.
>
> In the above comments, removed 'toplevel' as we track memory usage for
> both toplevel and subtransactions.
Correct.
>
> 3. There were still a few mentions of streaming which I have removed.
>
ok
> 4. In the docs, the type for stats spill_* was integer whereas it
> should be bigint.
ok
>
> 5.
> +UpdateSpillStats(LogicalDecodingContext *ctx)
> +{
> + ReorderBuffer *rb = ctx->reorder;
> +
> + SpinLockAcquire(&MyWalSnd->mutex);
> +
> + MyWalSnd->spillTxns = rb->spillTxns;
> + MyWalSnd->spillCount = rb->spillCount;
> + MyWalSnd->spillBytes = rb->spillBytes;
> +
> + elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld",
> + rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
>
> Changed the above elog to DEBUG1 as otherwise it was getting printed
> very frequently. I think we can make it DEBUG2 if we want.
Yeah, it should not be WARNING.
>
> 6. There was an extra space in rules.out due to which test was
> failing. I have fixed it.
My Bad. I have induced while separating out the changes for the spilling.
> What do you think?
I have reviewed your changes and looks fine to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-07 11:43 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-07 11:43 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 7, 2019 at 3:50 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Nov 7, 2019 at 3:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > What do you think?
> I have reviewed your changes and looks fine to me.
>
Okay, thanks. I am also happy with the two patches I have posted in
my last email [1].
Tomas, would you like to take a look at those patches and commit them
if you are happy or would you like me to do the same?
Some notes before commit:
--------------------------------------
1.
Commit message need to be changed for the first patch
-------------------------------------------------------------------------
A.
> The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this
SET logical_decoding_work_mem = '128kB'
> to trigger very aggressive streaming. The minimum value is 64kB.
I think this patch doesn't contain streaming, so we either need to
reword it or remove it.
B.
> The logical_decoding_work_mem may be set either in postgresql.conf, in which case it serves as the default for all publishers on that instance, or when creating the
> subscription, using a work_mem paramemter in the WITH clause (specifies number of kilobytes).
We need to reword this as we have decided to remove the setting from
the subscription side as of now.
2. I think we can change the message level in UpdateSpillStats() to DEBUG2.
3. I think we need catversion bump for the second patch.
4. I think we can combine both patches and commit as one patch, but it
is okay to commit them separately as well.
[1] - https://www.postgresql.org/message-id/CAA4eK1Kdmi6VVguKEHV6Ho2isCPVFdQtt0WLsK10fiuE59_0Yw%40mail.gma...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-12 10:42 Alexey Kondratov <a.kondratov@postgrespro.ru>
parent: Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-11-12 10:42 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 04.11.2019 13:05, Kuntal Ghosh wrote:
> On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> So your result shows that with "streaming on", performance is
>> degrading? By any chance did you try to see where is the bottleneck?
>>
> Right. But, as we increase the logical_decoding_work_mem, the
> performance improves. I've not analyzed the bottleneck yet. I'm
> looking into the same.
My guess is that 64 kB is just too small value. In the table schema used
for tests every rows takes at least 24 bytes for storing column values.
Thus, with this logical_decoding_work_mem value the limit should be hit
after about 2500+ rows, or about 400 times during transaction of 1000000
rows size.
It is just too frequent, while ReorderBufferStreamTXN includes a whole
bunch of logic, e.g. it always starts internal transaction:
/*
* Decoding needs access to syscaches et al., which in turn use
* heavyweight locks and such. Thus we need to have enough state around to
* keep track of those. The easiest way is to simply use a transaction
* internally. That also allows us to easily enforce that nothing writes
* to the database by checking for xid assignments. ...
*/
Also it issues separated stream_start/stop messages around each streamed
transaction chunk. So if streaming starts and stops too frequently it
adds additional overhead and may even interfere with current in-progress
transaction.
If I get it correctly, then it is rather expected with too small values
of logical_decoding_work_mem. Probably it may be optimized, but I am not
sure that it is worth doing right now.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-12 11:34 Kuntal Ghosh <kuntalghosh.2007@gmail.com>
parent: Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 0 replies; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-12 11:34 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Nov 12, 2019 at 4:12 PM Alexey Kondratov
<a.kondratov@postgrespro.ru> wrote:
>
> On 04.11.2019 13:05, Kuntal Ghosh wrote:
> > On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >> So your result shows that with "streaming on", performance is
> >> degrading? By any chance did you try to see where is the bottleneck?
> >>
> > Right. But, as we increase the logical_decoding_work_mem, the
> > performance improves. I've not analyzed the bottleneck yet. I'm
> > looking into the same.
>
> My guess is that 64 kB is just too small value. In the table schema used
> for tests every rows takes at least 24 bytes for storing column values.
> Thus, with this logical_decoding_work_mem value the limit should be hit
> after about 2500+ rows, or about 400 times during transaction of 1000000
> rows size.
>
> It is just too frequent, while ReorderBufferStreamTXN includes a whole
> bunch of logic, e.g. it always starts internal transaction:
>
> /*
> * Decoding needs access to syscaches et al., which in turn use
> * heavyweight locks and such. Thus we need to have enough state around to
> * keep track of those. The easiest way is to simply use a transaction
> * internally. That also allows us to easily enforce that nothing writes
> * to the database by checking for xid assignments. ...
> */
>
> Also it issues separated stream_start/stop messages around each streamed
> transaction chunk. So if streaming starts and stops too frequently it
> adds additional overhead and may even interfere with current in-progress
> transaction.
>
Yeah, I've also found the same. With stream_start/stop message, it
writes 1 byte of checksum and 4 bytes of number of sub-transactions
which increases the write amplification significantly.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-13 12:25 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-13 12:25 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
As mentioned by me a few days back that the first patch in this series
is ready to go [1] (I am hoping Tomas will pick it up), so I have
started the review of other patches
Review/Questions on 0002-Immediately-WAL-log-assignments.patch
-------------------------------------------------------------------------------------------------
1. This patch adds the top_xid in WAL whenever the first time WAL for
a subtransaction XID is written to correctly decode the changes of
in-progress transaction. This patch also removes logging and applying
WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
of that, it prunes KnownAssignedXids to prevent overflow of that
array. See comments in procarray.c (KnownAssignedTransactionIds
sub-module). Can you please explain how after removing the WAL for
XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
and there is no impact of same?
2.
+#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
This doesn't seem to be used in this patch.
[1] - https://www.postgresql.org/message-id/CAA4eK1JM0%3DRwODZQrn8DTQ3dbcb9xwKDdHCmVOryAk_xoKf9Nw%40mail.g...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-14 04:07 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-14 04:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
>
> As mentioned by me a few days back that the first patch in this series
> is ready to go [1] (I am hoping Tomas will pick it up), so I have
> started the review of other patches
>
> Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> -------------------------------------------------------------------------------------------------
> 1. This patch adds the top_xid in WAL whenever the first time WAL for
> a subtransaction XID is written to correctly decode the changes of
> in-progress transaction. This patch also removes logging and applying
> WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> of that, it prunes KnownAssignedXids to prevent overflow of that
> array. See comments in procarray.c (KnownAssignedTransactionIds
> sub-module). Can you please explain how after removing the WAL for
> XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> and there is no impact of same?
It seems like a problem to me as well. One option could be that
since now we are adding the top transaction id in the first WAL of the
subtransaction we can directly update the pg_subtrans and avoid adding
sub transaction id in the KnownAssignedXids and mark it as
lastOverflowedXid. But, I don't think we should go in that direction
otherwise it will impact the performance of visibility check on the
hot-standby. Let's see what Tomas has in mind.
>
> 2.
> +#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
>
> This doesn't seem to be used in this patch.
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-14 06:40 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-14 06:40 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 9:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> >
> > As mentioned by me a few days back that the first patch in this series
> > is ready to go [1] (I am hoping Tomas will pick it up), so I have
> > started the review of other patches
> >
> > Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> > -------------------------------------------------------------------------------------------------
> > 1. This patch adds the top_xid in WAL whenever the first time WAL for
> > a subtransaction XID is written to correctly decode the changes of
> > in-progress transaction. This patch also removes logging and applying
> > WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> > of that, it prunes KnownAssignedXids to prevent overflow of that
> > array. See comments in procarray.c (KnownAssignedTransactionIds
> > sub-module). Can you please explain how after removing the WAL for
> > XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> > and there is no impact of same?
>
> It seems like a problem to me as well. One option could be that
> since now we are adding the top transaction id in the first WAL of the
> subtransaction we can directly update the pg_subtrans and avoid adding
> sub transaction id in the KnownAssignedXids and mark it as
> lastOverflowedXid.
>
Hmm, I am not sure if we can do that easily because I think in
RecordKnownAssignedTransactionIds, we add those based on the gap via
KnownAssignedXidsAdd and only remove them later while applying WAL for
XLOG_XACT_ASSIGNMENT. I think if we really want to go in this
direction then for each WAL record we need to check if it has
XLR_BLOCK_ID_TOPLEVEL_XID set and then call function
ProcArrayApplyXidAssignment() with the required information. I think
this line of attack has WAL overhead both on master whenever
subtransactions are involved and also on hot-standby for doing the
work for each subtransaction separately. The WAL apply needs to
acquire and release PROCArrayLock in exclusive mode for each
subtransaction whereas now it does it once for
PGPROC_MAX_CACHED_SUBXIDS number of subtransactions which can conflict
with queries running on standby.
The other idea could be that we keep the current XLOG_XACT_ASSIGNMENT
mechanism (WAL logging and apply of same on hot-standby) as it is and
additionally log top_xid the first time when WAL is written for a
subtransaction only when wal_level >= WAL_LEVEL_LOGICAL. Then use the
same for logical decoding. The advantage of this approach is that we
will incur the overhead of additional transactionid only when required
especially not with default server configuration.
Thoughts?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-14 10:09 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-14 10:09 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 12:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Nov 14, 2019 at 9:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > >
> > > As mentioned by me a few days back that the first patch in this series
> > > is ready to go [1] (I am hoping Tomas will pick it up), so I have
> > > started the review of other patches
> > >
> > > Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> > > -------------------------------------------------------------------------------------------------
> > > 1. This patch adds the top_xid in WAL whenever the first time WAL for
> > > a subtransaction XID is written to correctly decode the changes of
> > > in-progress transaction. This patch also removes logging and applying
> > > WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> > > of that, it prunes KnownAssignedXids to prevent overflow of that
> > > array. See comments in procarray.c (KnownAssignedTransactionIds
> > > sub-module). Can you please explain how after removing the WAL for
> > > XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> > > and there is no impact of same?
> >
> > It seems like a problem to me as well. One option could be that
> > since now we are adding the top transaction id in the first WAL of the
> > subtransaction we can directly update the pg_subtrans and avoid adding
> > sub transaction id in the KnownAssignedXids and mark it as
> > lastOverflowedXid.
> >
>
> Hmm, I am not sure if we can do that easily because I think in
> RecordKnownAssignedTransactionIds, we add those based on the gap via
> KnownAssignedXidsAdd and only remove them later while applying WAL for
> XLOG_XACT_ASSIGNMENT. I think if we really want to go in this
> direction then for each WAL record we need to check if it has
> XLR_BLOCK_ID_TOPLEVEL_XID set and then call function
> ProcArrayApplyXidAssignment() with the required information. I think
> this line of attack has WAL overhead both on master whenever
> subtransactions are involved and also on hot-standby for doing the
> work for each subtransaction separately. The WAL apply needs to
> acquire and release PROCArrayLock in exclusive mode for each
> subtransaction whereas now it does it once for
> PGPROC_MAX_CACHED_SUBXIDS number of subtransactions which can conflict
> with queries running on standby.
Right
>
> The other idea could be that we keep the current XLOG_XACT_ASSIGNMENT
> mechanism (WAL logging and apply of same on hot-standby) as it is and
> additionally log top_xid the first time when WAL is written for a
> subtransaction only when wal_level >= WAL_LEVEL_LOGICAL. Then use the
> same for logical decoding. The advantage of this approach is that we
> will incur the overhead of additional transactionid only when required
> especially not with default server configuration.
>
> Thoughts?
The idea seems reasonable to me.
Apart from this, I have another question in
0003-Issue-individual-invalidations-with-wal_level-logical.patch
@@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
{
AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.sn.id = SHAREDINVALSNAPSHOT_ID;
+ msg.sn.dbId = dbId;
+ msg.sn.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
I am not sure why do we need to explicitly WAL log the snapshot
invalidation? because this is logged for invalidating the catalog
snapshot and for logical decoding we use HistoricSnapshot, not the
catalog snapshot. I might be missing something?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-15 10:20 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-15 10:20 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 3:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
>
> Apart from this, I have another question in
> 0003-Issue-individual-invalidations-with-wal_level-logical.patch
>
> @@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
> {
> AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
> dbId, relId);
> +
> + /* Issue an invalidation WAL record (when wal_level=logical) */
> + if (XLogLogicalInfoActive())
> + {
> + SharedInvalidationMessage msg;
> +
> + msg.sn.id = SHAREDINVALSNAPSHOT_ID;
> + msg.sn.dbId = dbId;
> + msg.sn.relId = relId;
> +
> + LogLogicalInvalidations(1, &msg, false);
> + }
> }
>
> I am not sure why do we need to explicitly WAL log the snapshot
> invalidation? because this is logged for invalidating the catalog
> snapshot and for logical decoding we use HistoricSnapshot, not the
> catalog snapshot.
>
I think it has been logged because without this patch as well we log
all the invalidation messages at commit time and process them during
decoding. However, I agree that this particular invalidation message
is not required for logical decoding for the reason you mentioned. I
think as we are explicitly logging invalidations, so it is better to
avoid this if we can.
Few other comments on this patch:
1.
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
Here, why are we executing messages individually? Can't we just
follow what we do in DecodeCommit which is to record the invalidations
in ReorderBufferTXN as we encounter them and then allow them to
execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
reason why we don't do ReorderBufferXidSetCatalogChanges when we
receive any invalidation message?
2.
@@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
ReorderBufferTXN *txn,
* although we don't check the memory limit when restoring the changes in
* this branch (we only do that when initially queueing the changes after
* decoding), because we will release the changes later, and that will
- * update the accounting too (subtracting the size from the counters).
- * And we don't want to underflow there.
+ * update the accounting too (subtracting the size from the counters). And
+ * we don't want to underflow there.
*/
This seems like an unrelated change.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-15 10:31 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-15 10:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Nov 14, 2019 at 3:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> >
> > Apart from this, I have another question in
> > 0003-Issue-individual-invalidations-with-wal_level-logical.patch
> >
> > @@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
> > {
> > AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
> > dbId, relId);
> > +
> > + /* Issue an invalidation WAL record (when wal_level=logical) */
> > + if (XLogLogicalInfoActive())
> > + {
> > + SharedInvalidationMessage msg;
> > +
> > + msg.sn.id = SHAREDINVALSNAPSHOT_ID;
> > + msg.sn.dbId = dbId;
> > + msg.sn.relId = relId;
> > +
> > + LogLogicalInvalidations(1, &msg, false);
> > + }
> > }
> >
> > I am not sure why do we need to explicitly WAL log the snapshot
> > invalidation? because this is logged for invalidating the catalog
> > snapshot and for logical decoding we use HistoricSnapshot, not the
> > catalog snapshot.
> >
>
> I think it has been logged because without this patch as well we log
> all the invalidation messages at commit time and process them during
> decoding. However, I agree that this particular invalidation message
> is not required for logical decoding for the reason you mentioned. I
> think as we are explicitly logging invalidations, so it is better to
> avoid this if we can.
Ok
>
> Few other comments on this patch:
> 1.
> + case REORDER_BUFFER_CHANGE_INVALIDATION:
> +
> + /*
> + * Execute the invalidation message locally.
> + *
> + * XXX Do we need to care about relcacheInitFileInval and
> + * the other fields added to ReorderBufferChange, or just
> + * about the message itself?
> + */
> + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> + break;
>
> Here, why are we executing messages individually? Can't we just
> follow what we do in DecodeCommit which is to record the invalidations
> in ReorderBufferTXN as we encounter them and then allow them to
> execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> reason why we don't do ReorderBufferXidSetCatalogChanges when we
> receive any invalidation message?
IMHO, the reason is that in DecodeCommit, we get all the invalidation
at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
know which invalidation message to execute so for being safe we have
to execute all. But, since we are logging all invalidation
individually, we exactly know at this stage which cache to invalidate.
So it is better to only invalidate required cache not all.
>
> 2.
> @@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
> ReorderBufferTXN *txn,
> * although we don't check the memory limit when restoring the changes in
> * this branch (we only do that when initially queueing the changes after
> * decoding), because we will release the changes later, and that will
> - * update the accounting too (subtracting the size from the counters).
> - * And we don't want to underflow there.
> + * update the accounting too (subtracting the size from the counters). And
> + * we don't want to underflow there.
> */
>
> This seems like an unrelated change.
Indeed.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-15 10:49 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-15 10:49 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Few other comments on this patch:
> > 1.
> > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > +
> > + /*
> > + * Execute the invalidation message locally.
> > + *
> > + * XXX Do we need to care about relcacheInitFileInval and
> > + * the other fields added to ReorderBufferChange, or just
> > + * about the message itself?
> > + */
> > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > + break;
> >
> > Here, why are we executing messages individually? Can't we just
> > follow what we do in DecodeCommit which is to record the invalidations
> > in ReorderBufferTXN as we encounter them and then allow them to
> > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > receive any invalidation message?
> IMHO, the reason is that in DecodeCommit, we get all the invalidation
> at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> know which invalidation message to execute so for being safe we have
> to execute all. But, since we are logging all invalidation
> individually, we exactly know at this stage which cache to invalidate.
> So it is better to only invalidate required cache not all.
>
In that case, invalidations can be processed multiple times, the first
time when these individual WAL logs for invalidation are processed and
then later at commit time when we accumulate all invalidation messages
and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
Can we avoid to execute invalidations from other places after this
patch which also includes executing them as part of XLOG_INVALIDATIONS
processing?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-16 13:14 Amit Kapila <amit.kapila16@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-16 13:14 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: vignesh C <vignesh21@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 7, 2019 at 5:13 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Some notes before commit:
> --------------------------------------
> 1.
> Commit message need to be changed for the first patch
> -------------------------------------------------------------------------
> A.
> > The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this
>
> SET logical_decoding_work_mem = '128kB'
>
> > to trigger very aggressive streaming. The minimum value is 64kB.
>
> I think this patch doesn't contain streaming, so we either need to
> reword it or remove it.
>
> B.
> > The logical_decoding_work_mem may be set either in postgresql.conf, in which case it serves as the default for all publishers on that instance, or when creating the
> > subscription, using a work_mem paramemter in the WITH clause (specifies number of kilobytes).
>
> We need to reword this as we have decided to remove the setting from
> the subscription side as of now.
>
> 2. I think we can change the message level in UpdateSpillStats() to DEBUG2.
>
I have made these modifications and additionally ran pgindent.
> 4. I think we can combine both patches and commit as one patch, but it
> is okay to commit them separately as well.
>
I am not sure if this is a good idea, so still kept them as separate.
Tomas, do let me know if you want to commit these or if you have any
comments, otherwise, I will commit these on Tuesday (19-Nov)?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.nov16.patch (application/octet-stream, 21.8K) download
From fe0f6a2ab4c7d00639e5b3c977837ab695fac1b1 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 16 Nov 2019 17:49:33 +0530
Subject: [PATCH 1/2] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage.
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to reduce the memory usage of walsenders or set the higher value to
reduce disk writes. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem can be set in postgresql.conf, in which
case it serves as the default for all publishers on that instance.
Author: Tomas Vondra, with changes by Dilip Kumar and Amit Kapila
Reviewed-by: Dilip Kumar and Amit Kapila
Tested-By: Vignesh C
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
---
contrib/test_decoding/logical.conf | 1 +
doc/src/sgml/config.sgml | 21 ++
src/backend/replication/logical/reorderbuffer.c | 293 +++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/reorderbuffer.h | 16 ++
6 files changed, 333 insertions(+), 12 deletions(-)
diff --git a/contrib/test_decoding/logical.conf b/contrib/test_decoding/logical.conf
index 367f706..07c4d3d 100644
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,2 +1,3 @@
wal_level = logical
max_replication_slots = 4
+logical_decoding_work_mem = 64kB
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f837703..d4d1fe4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1732,6 +1732,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are written to local disk. This
+ limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 62e5424..d82a5f1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each transaction. When the total amount of used memory exceeds the
+ * limit, the transaction consuming the most memory is then serialized to
+ * disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when reaching
+ * memory limit. So we just don't count them, because it might easily
+ * trigger a pointless attempt to spill.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2103,6 +2196,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.cmax = cmax;
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
+ change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
dlist_push_tail(&txn->tuplecids, &change->node);
@@ -2230,20 +2324,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2671,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3020,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters). And
+ * we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3249,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3282,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going to modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the change
+ * now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3438,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 4b3769b..ba4edde 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -66,6 +66,7 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
#include "replication/walreceiver.h"
@@ -2257,6 +2258,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ 65536, 64, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index be02a76..46a06ff 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 64kB
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..7c94d92 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
--
1.8.3.1
0002-Track-statistics-for-spilling-of-changes-from-Reorde.nov16.patch (application/octet-stream, 11.5K) download
From e5191f045484d5cd27578868598dec94fcbc06db Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 16 Nov 2019 18:24:00 +0530
Subject: [PATCH 2/2] Track statistics for spilling of changes from
ReorderBuffer.
This adds the statistics about transactions spilled to disk from
ReorderBuffer. Users can query the pg_stat_replication view to check
these stats.
Author: Tomas Vondra, with bug-fixes and minor changes by Dilip Kumar
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
---
doc/src/sgml/monitoring.sgml | 20 ++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 12 +++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 901fee9..a3c5f86 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1972,6 +1972,26 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<entry><type>timestamp with time zone</type></entry>
<entry>Send time of last reply message received from standby server</entry>
</row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_decoding_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.</entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4456fef..f7800f0 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index d82a5f1..53affeb 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2415,6 +2419,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2473,6 +2478,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillBytes += size;
+
+ /* Don't consider already serialized transaction. */
+ rb->spillTxns += txn->serialized ? 0 : 1;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 7f56715..fa75872 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2318,6 +2325,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3219,7 +3229,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3274,6 +3284,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3294,6 +3307,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3375,6 +3391,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3611,3 +3632,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7c94d92..0867ee9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is why we keep
+ * two different counters. For spilling, the transaction counter includes
+ * both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 14e7214..22e6c86 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1952,9 +1952,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-18 11:32 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-18 11:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Few other comments on this patch:
> > > 1.
> > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > +
> > > + /*
> > > + * Execute the invalidation message locally.
> > > + *
> > > + * XXX Do we need to care about relcacheInitFileInval and
> > > + * the other fields added to ReorderBufferChange, or just
> > > + * about the message itself?
> > > + */
> > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > + break;
> > >
> > > Here, why are we executing messages individually? Can't we just
> > > follow what we do in DecodeCommit which is to record the invalidations
> > > in ReorderBufferTXN as we encounter them and then allow them to
> > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > receive any invalidation message?
I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
commit. Because this is required to add any committed transaction to
the snapshot if it has done any catalog changes. So I think there is
no point in setting that flag every time we get an invalidation
message.
> > IMHO, the reason is that in DecodeCommit, we get all the invalidation
> > at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> > know which invalidation message to execute so for being safe we have
> > to execute all. But, since we are logging all invalidation
> > individually, we exactly know at this stage which cache to invalidate.
> > So it is better to only invalidate required cache not all.
> >
>
> In that case, invalidations can be processed multiple times, the first
> time when these individual WAL logs for invalidation are processed and
> then later at commit time when we accumulate all invalidation messages
> and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
> Can we avoid to execute invalidations from other places after this
> patch which also includes executing them as part of XLOG_INVALIDATIONS
> processing?
I think we can avoid invalidation which is done as part of
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. I need to further
investigate the invalidation which is done as part of
XLOG_INVALIDATIONS.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-19 11:53 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-19 11:53 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Few other comments on this patch:
> > > > 1.
> > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > +
> > > > + /*
> > > > + * Execute the invalidation message locally.
> > > > + *
> > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > + * the other fields added to ReorderBufferChange, or just
> > > > + * about the message itself?
> > > > + */
> > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > + break;
> > > >
> > > > Here, why are we executing messages individually? Can't we just
> > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > receive any invalidation message?
>
> I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> commit. Because this is required to add any committed transaction to
> the snapshot if it has done any catalog changes.
>
Hmm, this is also used to build cid hash map (see
ReorderBufferBuildTupleCidHash) which we need to use while streaming
changes for the in-progress transactions. So, I think that it would
be required earlier (before commit) as well.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-19 11:55 Amit Kapila <amit.kapila16@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-19 11:55 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: vignesh C <vignesh21@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Nov 16, 2019 at 6:44 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Nov 7, 2019 at 5:13 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Some notes before commit:
> > --------------------------------------
> > 1.
> > Commit message need to be changed for the first patch
> > -------------------------------------------------------------------------
> > A.
> > > The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this
> >
> > SET logical_decoding_work_mem = '128kB'
> >
> > > to trigger very aggressive streaming. The minimum value is 64kB.
> >
> > I think this patch doesn't contain streaming, so we either need to
> > reword it or remove it.
> >
> > B.
> > > The logical_decoding_work_mem may be set either in postgresql.conf, in which case it serves as the default for all publishers on that instance, or when creating the
> > > subscription, using a work_mem paramemter in the WITH clause (specifies number of kilobytes).
> >
> > We need to reword this as we have decided to remove the setting from
> > the subscription side as of now.
> >
> > 2. I think we can change the message level in UpdateSpillStats() to DEBUG2.
> >
>
> I have made these modifications and additionally ran pgindent.
>
> > 4. I think we can combine both patches and commit as one patch, but it
> > is okay to commit them separately as well.
> >
>
> I am not sure if this is a good idea, so still kept them as separate.
>
I have committed the first patch. I will commit the second one
related to stats of spilled xacts on Thursday. The second patch needs
catalog version bump as well because we are modifying the catalog
contents in that patch.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-20 05:45 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-20 05:45 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Few other comments on this patch:
> > > > > 1.
> > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > +
> > > > > + /*
> > > > > + * Execute the invalidation message locally.
> > > > > + *
> > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > + * about the message itself?
> > > > > + */
> > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > + break;
> > > > >
> > > > > Here, why are we executing messages individually? Can't we just
> > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > receive any invalidation message?
> >
> > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > commit. Because this is required to add any committed transaction to
> > the snapshot if it has done any catalog changes.
> >
>
> Hmm, this is also used to build cid hash map (see
> ReorderBufferBuildTupleCidHash) which we need to use while streaming
> changes for the in-progress transactions. So, I think that it would
> be required earlier (before commit) as well.
>
Oh right, I guess I missed that part.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-20 14:52 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-20 14:52 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Few other comments on this patch:
> > > > > > 1.
> > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > +
> > > > > > + /*
> > > > > > + * Execute the invalidation message locally.
> > > > > > + *
> > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > + * about the message itself?
> > > > > > + */
> > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > + break;
> > > > > >
> > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > receive any invalidation message?
> > >
> > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > commit. Because this is required to add any committed transaction to
> > > the snapshot if it has done any catalog changes.
> > >
> >
> > Hmm, this is also used to build cid hash map (see
> > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > changes for the in-progress transactions. So, I think that it would
> > be required earlier (before commit) as well.
> >
> Oh right, I guess I missed that part.
Attached a new rebased version of the patch set. I have fixed all
the issues discussed up-thread and agreed upon.
Pending Issues:
1. The default value of the logical_decoding_work_mem is set to 64kb
in test_decoding/logical.conf. So we need to change the expected
output files for the test decoding module.
2. Need to complete the patch for concurrent abort handling of the
(sub)transaction. There are some pending issues with the existing
patch[1].
[1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0002-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 33439d94691e636a4439d4e234d76fc9916b8359 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH 02/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8fe38c3..4a853f3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 7f24f0c..4ef2661 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1071,6 +1071,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1109,6 +1110,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 42b76cb..0c7daf4 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
0001-Track-statistics-for-spilling-of-changes-from-Reorde.patch (application/octet-stream, 11.5K) download
From 45bdc8b9116844c2a458110363b14e820911bccb Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 16 Nov 2019 18:24:00 +0530
Subject: [PATCH 01/13] Track statistics for spilling of changes from
ReorderBuffer.
This adds the statistics about transactions spilled to disk from
ReorderBuffer. Users can query the pg_stat_replication view to check
these stats.
Author: Tomas Vondra, with bug-fixes and minor changes by Dilip Kumar
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
---
doc/src/sgml/monitoring.sgml | 20 ++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 12 +++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 901fee9..a3c5f86 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1972,6 +1972,26 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<entry><type>timestamp with time zone</type></entry>
<entry>Send time of last reply message received from standby server</entry>
</row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_decoding_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.</entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4456fef..f7800f0 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index d82a5f1..53affeb 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2415,6 +2419,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2473,6 +2478,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillBytes += size;
+
+ /* Don't consider already serialized transaction. */
+ rb->spillTxns += txn->serialized ? 0 : 1;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 7f56715..fa75872 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2318,6 +2325,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3219,7 +3229,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3274,6 +3284,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3294,6 +3307,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3375,6 +3391,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3611,3 +3632,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7c94d92..0867ee9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is why we keep
+ * two different counters. For spilling, the transaction counter includes
+ * both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index abe3a43..c9cc569 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1952,9 +1952,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0003-Issue-individual-invalidations-with-wal_level-logica.patch (application/octet-stream, 15.8K) download
From e4d3e144b8babaf08be669c17b577c5348e0e3cc Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH 03/13] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 4a853f3..a42d11f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 0c7daf4..5bd3893 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch (application/octet-stream, 8.1K) download
From d750bf1746a2cd61ff9d72ff690baf400512605c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From f2d26607b79cacc54c89b70a8f88d6194e814f26 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
0006-Gracefully-handle-concurrent-aborts-of-uncommitted-t.patch (application/octet-stream, 12.5K) download
From fc83f3bbe01cd7e807033e5a9b2ef8dd48b8f5ca Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH 06/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 0251050a1ba6ece6ee3a8826b5e1ac5a136c8e39 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH 07/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 537e681..76a105a 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
0008-Support-logical_decoding_work_mem-set-from-create-su.patch (application/octet-stream, 13.1K) download
From 54556711ea38bcd407bdc65f12f6c70ec2e8a592 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH 08/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d4d1fe4..f449fa1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303..14c0ce8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1726,6 +1726,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
0010-Track-statistics-for-streaming.patch (application/octet-stream, 11.6K) download
From bf0e4ba0c0441cbbf52989f35b49fdbf7da01fdc Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 12:19:49 +0530
Subject: [PATCH 10/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++++
src/backend/catalog/system_views.sql | 5 ++++-
src/backend/replication/logical/reorderbuffer.c | 13 +++++++++++
src/backend/replication/walsender.c | 30 ++++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 +++++++----
src/include/replication/walsender_private.h | 5 +++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 89 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a2ae283..8619a01 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1269,7 +1269,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1290,7 +1290,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2334,6 +2335,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3235,7 +3239,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3293,6 +3297,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3316,6 +3323,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3402,6 +3412,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3650,8 +3665,13 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
- rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fa0a2a1..9a508bf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0009-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 81193e00db98be3807352d84e712d13b5b4049d9 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH 09/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 1f8821c..9307b67 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 14c0ce8..65e47aa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1099,6 +1568,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1547,6 +2019,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1727,6 +2757,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fa75872..a2ae283 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -945,6 +945,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
0012-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1010B) download
From 9699ab41ef32af4a0bafff7d4263a9d6d534ccc4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 12/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
0011-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From cf6ce6192de8ebcaa8942fb234426e30e976055d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH 11/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
0013-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 79bba6bd97b938917720012d99a19ec614752976 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 13/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-21 03:32 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-21 03:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 20, 2019 at 8:22 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > > Few other comments on this patch:
> > > > > > > 1.
> > > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > > +
> > > > > > > + /*
> > > > > > > + * Execute the invalidation message locally.
> > > > > > > + *
> > > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > > + * about the message itself?
> > > > > > > + */
> > > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > > + break;
> > > > > > >
> > > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > > receive any invalidation message?
> > > >
> > > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > > commit. Because this is required to add any committed transaction to
> > > > the snapshot if it has done any catalog changes.
> > > >
> > >
> > > Hmm, this is also used to build cid hash map (see
> > > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > > changes for the in-progress transactions. So, I think that it would
> > > be required earlier (before commit) as well.
> > >
> > Oh right, I guess I missed that part.
>
> Attached a new rebased version of the patch set. I have fixed all
> the issues discussed up-thread and agreed upon.
>
> Pending Issues:
> 1. The default value of the logical_decoding_work_mem is set to 64kb
> in test_decoding/logical.conf. So we need to change the expected
> output files for the test decoding module.
> 2. Need to complete the patch for concurrent abort handling of the
> (sub)transaction. There are some pending issues with the existing
> patch[1].
> [1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
Apart from these there is one more issue reported upthread[2]
[2] https://www.postgresql.org/message-id/CAFiTN-vrSNkAfRVrWKe2R1dqFBTubjt%3DDYS%3DjhH%2BjiCoBODdaw%40ma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-21 10:44 Amit Kapila <amit.kapila16@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-11-21 10:44 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: vignesh C <vignesh21@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Nov 19, 2019 at 5:25 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Nov 16, 2019 at 6:44 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Nov 7, 2019 at 5:13 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Some notes before commit:
> > > --------------------------------------
> > > 1.
> > > Commit message need to be changed for the first patch
> > > -------------------------------------------------------------------------
> > > A.
> > > > The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this
> > >
> > > SET logical_decoding_work_mem = '128kB'
> > >
> > > > to trigger very aggressive streaming. The minimum value is 64kB.
> > >
> > > I think this patch doesn't contain streaming, so we either need to
> > > reword it or remove it.
> > >
> > > B.
> > > > The logical_decoding_work_mem may be set either in postgresql.conf, in which case it serves as the default for all publishers on that instance, or when creating the
> > > > subscription, using a work_mem paramemter in the WITH clause (specifies number of kilobytes).
> > >
> > > We need to reword this as we have decided to remove the setting from
> > > the subscription side as of now.
> > >
> > > 2. I think we can change the message level in UpdateSpillStats() to DEBUG2.
> > >
> >
> > I have made these modifications and additionally ran pgindent.
> >
> > > 4. I think we can combine both patches and commit as one patch, but it
> > > is okay to commit them separately as well.
> > >
> >
> > I am not sure if this is a good idea, so still kept them as separate.
> >
>
> I have committed the first patch. I will commit the second one
> related to stats of spilled xacts on Thursday. The second patch needs
> catalog version bump as well because we are modifying the catalog
> contents in that patch.
>
Committed the second one as well. Now, we can move to a review of
patches for "streaming of in-progress transactions".
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-11-22 07:48 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-22 07:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 21, 2019 at 9:02 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 20, 2019 at 8:22 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > > > >
> > > > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > Few other comments on this patch:
> > > > > > > > 1.
> > > > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > > > +
> > > > > > > > + /*
> > > > > > > > + * Execute the invalidation message locally.
> > > > > > > > + *
> > > > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > > > + * about the message itself?
> > > > > > > > + */
> > > > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > > > + break;
> > > > > > > >
> > > > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > > > receive any invalidation message?
> > > > >
> > > > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > > > commit. Because this is required to add any committed transaction to
> > > > > the snapshot if it has done any catalog changes.
> > > > >
> > > >
> > > > Hmm, this is also used to build cid hash map (see
> > > > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > > > changes for the in-progress transactions. So, I think that it would
> > > > be required earlier (before commit) as well.
> > > >
> > > Oh right, I guess I missed that part.
> >
> > Attached a new rebased version of the patch set. I have fixed all
> > the issues discussed up-thread and agreed upon.
> >
> > Pending Issues:
> > 1. The default value of the logical_decoding_work_mem is set to 64kb
> > in test_decoding/logical.conf. So we need to change the expected
> > output files for the test decoding module.
> > 2. Need to complete the patch for concurrent abort handling of the
> > (sub)transaction. There are some pending issues with the existing
> > patch[1].
> > [1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
> Apart from these there is one more issue reported upthread[2]
> [2] https://www.postgresql.org/message-id/CAFiTN-vrSNkAfRVrWKe2R1dqFBTubjt%3DDYS%3DjhH%2BjiCoBODdaw%40ma...
>
I have rebased the patch on the latest head and also fix the issue of
"concurrent abort handling of the (sub)transaction." and attached as
(v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
the complete patch set. I have added the version number so that we
can track the changes.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v1-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 949e0782796eef008afd703c85a8eaca6625709d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v1 01/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8fe38c3..4a853f3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 7f24f0c..4ef2661 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1071,6 +1071,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1109,6 +1110,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v1-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.8K) download
From 4ddd0a234ec138105302d62bb9900a8c858a225e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v1 02/13] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 4a853f3..a42d11f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v1-0004-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From ad5a84eef45bc215044748f382e0d5d586a502be Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v1 04/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v1-0005-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 12.5K) download
From 193f2812afa906a07cc5f90a8b872cebbf96402b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v1 05/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v1-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.9K) download
From d3e02ec7f0485638247e236937e25d62a1f44ab3 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v1 03/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v1-0006-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 95cc2862df84b5bc02ca7ac0a3ab8bd2f95f52f2 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v1 06/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v1-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From ac49838b249e181e0fbd9297b034b96b6636f964 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v1 07/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d4d1fe4..f449fa1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303..14c0ce8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1726,6 +1726,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v1-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 3968b2aa385b703316109df3eaef0dce7e027f0b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH v1 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 1f8821c..9307b67 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 14c0ce8..65e47aa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1099,6 +1568,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1547,6 +2019,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1727,6 +2757,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fa75872..a2ae283 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -945,6 +945,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v1-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.6K) download
From 9ebaada8dc1630ba7b039da109f14f3eb111ca50 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 12:19:49 +0530
Subject: [PATCH v1 09/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++++
src/backend/catalog/system_views.sql | 5 ++++-
src/backend/replication/logical/reorderbuffer.c | 13 +++++++++++
src/backend/replication/walsender.c | 30 ++++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 +++++++----
src/include/replication/walsender_private.h | 5 +++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 89 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a2ae283..8619a01 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1269,7 +1269,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1290,7 +1290,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2334,6 +2335,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3235,7 +3239,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3293,6 +3297,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3316,6 +3323,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3402,6 +3412,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3650,8 +3665,13 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
- rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d4fa6dd..b3bbc82 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v1-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 989a8c4e3f0574949f36e2ecaa685548aa7726b1 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v1 10/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v1-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 5bfee24870b38fb1b60b1c0806f98fd289edabc6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v1 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v1-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 739e742dfe7b453857d6ef4d354cfbfe1d25c111 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v1 12/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v1-0013-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 4.0K) download
From 39df08b4bbe216f2fe1a386c5afd63775565099a Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v1 13/13] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 38 +++++++++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 410da36..710a22f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2350,9 +2350,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3281,6 +3281,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3411,6 +3412,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3472,6 +3480,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3654,6 +3666,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+
+ /* Invalidate current schema as well */
txn->is_schema_sent = false;
break;
@@ -3717,6 +3731,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3735,7 +3752,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c183fce..1db6da6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -242,6 +242,11 @@ typedef struct ReorderBufferTXN
bool is_schema_sent;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-01 02:28 Michael Paquier <michael@paquier.xyz>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Michael Paquier @ 2019-12-01 02:28 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> I have rebased the patch on the latest head and also fix the issue of
> "concurrent abort handling of the (sub)transaction." and attached as
> (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> the complete patch set. I have added the version number so that we
> can track the changes.
The patch has rotten a bit and does not apply anymore. Could you
please send a rebased version? I have moved it to next CF, waiting on
author.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-02 08:31 Dilip Kumar <dilipbalaut@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 5 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-12-02 08:31 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > I have rebased the patch on the latest head and also fix the issue of
> > "concurrent abort handling of the (sub)transaction." and attached as
> > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > the complete patch set. I have added the version number so that we
> > can track the changes.
>
> The patch has rotten a bit and does not apply anymore. Could you
> please send a rebased version? I have moved it to next CF, waiting on
> author.
I have rebased the patch set on the latest head.
Apart from this, there is one issue reported by my colleague Vignesh.
The issue is that if we use more than two relations in a transaction
then there is an error on standby (no relation map entry for remote
relation ID 16390). After analyzing I have found that for the
streaming transaction an "is_schema_sent" flag is kept in
ReorderBufferTXN. And, I think that is done so that we can send the
schema for each transaction stream so that if any subtransaction gets
aborted we don't lose the logical WAL for that schema. But, this
solution has induced a very basic issue that if a transaction operate
on more than 1 relation then after sending the schema for the first
relation it will mark the flag true and the schema for the subsequent
relations will never be sent. I am still working on finding a better
solution for this if anyone has any opinion/solution about this feel
free to suggest.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v2-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 05fe8c63cc2e81b32aaf72599190fa4521362429 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v2 01/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6a..708e523 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b0..4435c63 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611..a676151 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v2-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.8K) download
From a10bea688773350e4262839d9d64df2ec2b7c429 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v2 02/13] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e523..da15556 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v2-0004-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From b6825f74dfd9809dc06feb16073eab5714a4a488 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v2 04/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v2-0005-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 12.5K) download
From 2b3a261b2c80aa84073c39d7be76a42d6e491195 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v2 05/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v2-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.9K) download
From 64fa40c143ef89e61590a45f93f8e3dc675af4f2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v2 03/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e..1e934d2 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From bd34ab5db8501a4a8ea06bcd53c4386fe660a7ae Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v2 06/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v2-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 8f7065a4c1347fcf27b89650ab4a6cc6e9d6d012 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v2 07/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4ec13f3..3c32686 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ced0d19..ebb976c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1742,6 +1742,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714ea..1db706a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v2-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 6aa8ffd27af999e878f1fd53f83d3a70dfbec0ac Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH v2 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 4643af9..15e7140 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ebb976c..62be080 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -540,6 +959,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -555,6 +977,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -590,6 +1015,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -693,6 +1121,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -813,6 +1244,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -912,6 +1346,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1003,6 +1440,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1100,6 +1553,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1115,6 +1584,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1563,6 +2035,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1743,6 +2773,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 8bafa65..0837264 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706a..3d19b5d 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v2-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From ebc31a6153b7a4e7887d5d8b7c6c4959c54f4e60 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH v2 09/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0837264..9f34b3b 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1292,7 +1292,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1313,7 +1313,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2356,6 +2357,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3178,7 +3182,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3236,6 +3240,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3259,6 +3266,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3345,6 +3355,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3592,12 +3607,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b..3b897a5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v2-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From a97bae3b74b77b77df1ca134f61a764e6cea1bd3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v2 10/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v2-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From e468e357589eb43acb2b1117f9b3ada7965c86e1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v2 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v2-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 24b2c427831ba1e268f834ea368f796b38a81ad1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v2 12/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v2-0013-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 4.0K) download
From a0b6c483c0180771e9fd0a21f13aa3ce924fb0db Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v2 13/13] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 38 +++++++++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 410da36..710a22f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2350,9 +2350,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3281,6 +3281,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3411,6 +3412,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3472,6 +3480,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3654,6 +3666,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+
+ /* Invalidate current schema as well */
txn->is_schema_sent = false;
break;
@@ -3717,6 +3731,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3735,7 +3752,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c183fce..1db6da6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -242,6 +242,11 @@ typedef struct ReorderBufferTXN
bool is_schema_sent;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-09 07:56 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
4 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-12-09 07:56 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 2, 2019 at 2:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > I have rebased the patch on the latest head and also fix the issue of
> > > "concurrent abort handling of the (sub)transaction." and attached as
> > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > the complete patch set. I have added the version number so that we
> > > can track the changes.
> >
> > The patch has rotten a bit and does not apply anymore. Could you
> > please send a rebased version? I have moved it to next CF, waiting on
> > author.
>
> I have rebased the patch set on the latest head.
>
I have review the patch set and here are few comments/questions
1.
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
Should we show the tuple in the streamed change like we do for the
pg_decode_change?
2. pg_logical_slot_get_changes_guts
It recreate the decoding slot [ctx =
CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
to false, should we pass a parameter to
pg_logical_slot_get_changes_guts saying whether we want streamed results or not
3.
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
I did not understand, how this change is relavent to this patch
4.
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
In which case, txn->is_schema_sent will be true, because at the end of
the stream in ReorderBufferExecuteInvalidations we are always setting
it false,
so while sending next stream it will always be false. That means we
never required snapshot_now variable in ReorderBufferTXN.
5.
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
*rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
Same as point 4, during decode time it will never be true.
6.
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
Commit_time and end_lsn is used in standby_feedback
7.
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
We can not rollback intermediate subtransaction without rollbacking
latest sub-transaction, so why do we need
to search in the array? It will always be the the last subxact no?
8.
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
Why feedback is sent for every change?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-10 04:22 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
4 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-10 04:22 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > I have rebased the patch on the latest head and also fix the issue of
> > > "concurrent abort handling of the (sub)transaction." and attached as
> > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > the complete patch set. I have added the version number so that we
> > > can track the changes.
> >
> > The patch has rotten a bit and does not apply anymore. Could you
> > please send a rebased version? I have moved it to next CF, waiting on
> > author.
>
> I have rebased the patch set on the latest head.
>
> Apart from this, there is one issue reported by my colleague Vignesh.
> The issue is that if we use more than two relations in a transaction
> then there is an error on standby (no relation map entry for remote
> relation ID 16390). After analyzing I have found that for the
> streaming transaction an "is_schema_sent" flag is kept in
> ReorderBufferTXN. And, I think that is done so that we can send the
> schema for each transaction stream so that if any subtransaction gets
> aborted we don't lose the logical WAL for that schema. But, this
> solution has induced a very basic issue that if a transaction operate
> on more than 1 relation then after sending the schema for the first
> relation it will mark the flag true and the schema for the subsequent
> relations will never be sent.
>
How about keeping a list of top-level xids in each RelationSyncEntry?
Basically, whenever we send the schema for any transaction, we note
that in RelationSyncEntry and at abort time we can remove xid from the
list. Now, whenever, we check whether to send schema for any
operation in a transaction, we will check if our xid is present in
that list for a particular RelationSyncEntry and take an action based
on that (if xid is present, then we won't send the schema, otherwise,
send it).
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-10 04:53 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-12-10 04:53 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Dec 10, 2019 at 9:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> > >
> > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > > I have rebased the patch on the latest head and also fix the issue of
> > > > "concurrent abort handling of the (sub)transaction." and attached as
> > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > > the complete patch set. I have added the version number so that we
> > > > can track the changes.
> > >
> > > The patch has rotten a bit and does not apply anymore. Could you
> > > please send a rebased version? I have moved it to next CF, waiting on
> > > author.
> >
> > I have rebased the patch set on the latest head.
> >
> > Apart from this, there is one issue reported by my colleague Vignesh.
> > The issue is that if we use more than two relations in a transaction
> > then there is an error on standby (no relation map entry for remote
> > relation ID 16390). After analyzing I have found that for the
> > streaming transaction an "is_schema_sent" flag is kept in
> > ReorderBufferTXN. And, I think that is done so that we can send the
> > schema for each transaction stream so that if any subtransaction gets
> > aborted we don't lose the logical WAL for that schema. But, this
> > solution has induced a very basic issue that if a transaction operate
> > on more than 1 relation then after sending the schema for the first
> > relation it will mark the flag true and the schema for the subsequent
> > relations will never be sent.
> >
>
> How about keeping a list of top-level xids in each RelationSyncEntry?
> Basically, whenever we send the schema for any transaction, we note
> that in RelationSyncEntry and at abort time we can remove xid from the
> list. Now, whenever, we check whether to send schema for any
> operation in a transaction, we will check if our xid is present in
> that list for a particular RelationSyncEntry and take an action based
> on that (if xid is present, then we won't send the schema, otherwise,
> send it).
The idea make sense to me. I will try to write a patch for this and test.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-11 11:52 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-11 11:52 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have review the patch set and here are few comments/questions
>
> 1.
> +static void
> +pg_decode_stream_change(LogicalDecodingContext *ctx,
> + ReorderBufferTXN *txn,
> + Relation relation,
> + ReorderBufferChange *change)
> +{
> + OutputPluginPrepareWrite(ctx, true);
> + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> + OutputPluginWrite(ctx, true);
> +}
>
> Should we show the tuple in the streamed change like we do for the
> pg_decode_change?
>
I think so. The patch shows the message in
pg_decode_stream_message(), so why to prohibit showing tuple here?
> 2. pg_logical_slot_get_changes_guts
> It recreate the decoding slot [ctx =
> CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> to false, should we pass a parameter to
> pg_logical_slot_get_changes_guts saying whether we want streamed results or not
>
CreateDecodingContext internally calls StartupDecodingContext which
sets the value of streaming based on if the plugin has provided
callbacks for streaming functions. Isn't that sufficient? Why do we
need additional parameters here?
> 3.
> + XLogRecPtr prev_lsn = InvalidXLogRecPtr;
> ReorderBufferChange *change;
> ReorderBufferChange *specinsert = NULL;
>
> @@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> Relation relation = NULL;
> Oid reloid;
>
> + /*
> + * Enforce correct ordering of changes, merged from multiple
> + * subtransactions. The changes may have the same LSN due to
> + * MULTI_INSERT xlog records.
> + */
> + if (prev_lsn != InvalidXLogRecPtr)
> + Assert(prev_lsn <= change->lsn);
> +
> + prev_lsn = change->lsn;
> I did not understand, how this change is relavent to this patch
>
This is just to ensure that changes are in LSN order. I think as we
are merging the changes before commit for streaming, it is good to
have such an Assertion for ReorderBufferStreamTXN. And, if we want
to have it in ReorderBufferStreamTXN, then there is no harm in keeping
it in ReorderBufferCommit() at least to keep the code consistent. Do
you see any problem with this?
> 4.
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn, command_id);
>
> In which case, txn->is_schema_sent will be true, because at the end of
> the stream in ReorderBufferExecuteInvalidations we are always setting
> it false,
> so while sending next stream it will always be false. That means we
> never required snapshot_now variable in ReorderBufferTXN.
>
You are probably right, but as discussed we need to change this part
of design/code (when to send schema changes) due to the issues
discovered. So, I think this part will anyway change when we fix that
problem.
> 5.
> @@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
> *rb, TransactionId xid,
> txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
>
> txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
> +
> + /*
> + * We read catalog changes from WAL, which are not yet sent, so
> + * invalidate current schema in order output plugin can resend
> + * schema again.
> + */
> + txn->is_schema_sent = false;
>
> Same as point 4, during decode time it will never be true.
>
Sure, my previous point's reply applies here as well.
> 6.
> + /* send fields */
> + pq_sendint64(out, commit_lsn);
> + pq_sendint64(out, txn->end_lsn);
> + pq_sendint64(out, txn->commit_time);
>
> Commit_time and end_lsn is used in standby_feedback
>
I don't understand what you mean by this. Can you be a bit more clear?
>
> 7.
> + /* FIXME optimize the search by bsearch on sorted data */
> + for (i = nsubxacts; i > 0; i--)
> + {
> + if (subxacts[i - 1].xid == subxid)
> + {
> + subidx = (i - 1);
> + found = true;
> + break;
> + }
> + }
> We can not rollback intermediate subtransaction without rollbacking
> latest sub-transaction, so why do we need
> to search in the array? It will always be the the last subxact no?
>
The same thing is already mentioned in the comments above this code
("XXX Or perhaps we can rely on the aborts to arrive in the reverse
order, i.e. from the inner-most subxact (when nested)? In which case
we could simply check the last element."). I think what you are
saying is probably right, but we can leave this as it is for now
because this is a minor optimization which can be done later as well
if required. However, if you see any correctness issue, then we can
discuss.
> 8.
> + /*
> + * send feedback to upstream
> + *
> + * XXX Probably should send a valid LSN. But which one?
> + */
> + send_feedback(InvalidXLogRecPtr, false, false);
>
> Why feedback is sent for every change?
>
I will study this part of the patch and let you know my opinion.
Few comments on this patch series:
0001-Immediately-WAL-log-assignments:
------------------------------------------------------------
The commit message still refers to the old design for this patch. I
think you need to modify the commit message as per the latest patch.
0002-Issue-individual-invalidations-with-wal_level-log
----------------------------------------------------------------------------
1.
xact_desc_invalidations(StringInfo buf,
{
..
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
You have removed logging for the above cache but forgot to remove its
reference from one of the places. Also, I think you need to add a
comment somewhere in inval.c to say why you are writing for WAL for
some types of invalidations and not for others?
0003-Extend-the-output-plugin-API-with-stream-methods
--------------------------------------------------------------------------------
1.
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
stream_message_cb is mentioned twice. It seems the second one is for truncate.
2.
size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
/size of the transaction size/size of the transaction
no need to mention size twice.
3.
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress
transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting.
The guc name used is wrong. /Similarly to/Similar to/
4.
stream_start_cb_wrapper()
{
..
+ /* state.report_location = apply_lsn; */
..
+ /* FIXME ctx->write_location = apply_lsn; */
..
}
See, if we can fix these and similar in the callback for the stop. I
think we don't have final_lsn till we commit/abort. Can we compute
before calling these API's?
0005-Gracefully-handle-concurrent-aborts-of-uncommitte
----------------------------------------------------------------------------------
1.
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup
from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
Spurious line change.
2. The commit message of this patch refers to Prepared transactions.
I think that needs to be changed.
0006-Implement-streaming-mode-in-ReorderBuffer
-------------------------------------------------------------------------
1.
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+
ReorderBuffer *rb,
+
ReorderBufferTXN
*txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+
ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+
ReorderBuffer *rb,
+
ReorderBufferStreamIterTXNState * state);
Do we really need to introduce new APIs for iterating over changes
from streamed transactions? Why can't we reuse the same API's as we
use for committed xacts?
2.
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
Please write some comments atop ReorderBufferStreamCommit.
3.
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
..
+ if (txn->snapshot_now
== NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+
Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id ==
InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+
subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+
(txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+
txn->base_snapshot = subtxn->base_snapshot;
The logic here seems to be correct, but I am not sure why it is not
considered to purge the base snapshot before assigning the subtxn's
snapshot and similarly, we have not purged snapshot for subtxn once we
are done with it. I think we can use
ReorderBufferTransferSnapToParent to replace part of the logic here.
Do you see any reason for doing things differently here?
4. In ReorderBufferStreamTXN, why do you need to use
ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
5. I see a lot of code similarity in ReorderBufferStreamTXN and
existing ReorderBufferCommit. I understand that there are some subtle
differences due to which we need to write this new function but can't
we encapsulate the specific parts of code in functions and then call
from both places. I am talking about code in different cases for
change->action.
6. + * Note: We never stream and serialize a transaction at the same time (e
/(e/(we
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-11 18:16 Robert Haas <robertmhaas@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
4 siblings, 2 replies; 536+ messages in thread
From: Robert Haas @ 2019-12-11 18:16 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 2, 2019 at 3:32 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> I have rebased the patch set on the latest head.
0001 looks like a clever approach, but are you sure it doesn't hurt
performance when many small XLOG records are being inserted? I think
XLogRecordAssemble() can get pretty hot in some workloads.
With regard to 0002, logging a separate WAL record for each
invalidation seems painful; I think most operations that generate
invalidations generate a bunch of them all at once. Perhaps you could
just queue up invalidations as they happen, and then force anything
that's been queued up to be emitted into WAL just before you emit any
WAL record that might need to be decoded.
Regarding 0005, it seems to me that this is no good:
+ errmsg("improper heap_getnext call")));
I think we should be using elog() rather than ereport() here, because
this should only happen if there's a bug in a logical decoding plugin.
At first, I thought maybe this should just be an Assert(), but since
there are third-party logical decoding plugins available, checking
this even in non-assert builds seems like a good idea. However, I
think making it translatable is overkill; users should never see this,
only developers.
I also think that the message is really bad, because it just tells you
did something bad. It gives no inkling as to why it was bad.
0006 contains lots of XXX comments that look like real issues. I guess
those need to be fixed. Also, why don't we do the thing that the
commit message for 0006 says we could "theoretically" do? I don't
understand why we need the k-way merge at all,
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
There is no reason to ever write an if statement that contains only an
Assert, and it's bad style. Write Assert(prev_lsn == InvalidXLogRecPtr
|| prev_lsn <= change->lsn), or better yet, use XLogRecPtrIsInvalid.
The purpose and mechanism of the is_schema_sent flag is not clear to
me. The word "schema" here seems to be being used to mean "snapshot,"
which is rather confusing.
I'm also somewhat unclear on what's happening here with invalidations.
Perhaps that's as much a defect in my understanding as it is
reflective of any problem with the patch, but I also don't see any
comments either in 0002 or later patches explaining the theory of
operation. If I've missed some, please point me in the right
direction. Hypothetically speaking, it seems to me that if you just
did InvalidateSystemCaches() every time the snapshot changed, you
wouldn't need anything else (unless we're concerned with
non-transactional invalidation messages like smgr and relmapper
invalidations; not quite sure how those are handled). And, on the
other hand, if we don't do InvalidateSystemCaches() every time the
snapshot changes, then I don't understand why this works now, even
without streaming.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-12 04:14 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-12-12 04:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have review the patch set and here are few comments/questions
> >
> > 1.
> > +static void
> > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > + ReorderBufferTXN *txn,
> > + Relation relation,
> > + ReorderBufferChange *change)
> > +{
> > + OutputPluginPrepareWrite(ctx, true);
> > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > + OutputPluginWrite(ctx, true);
> > +}
> >
> > Should we show the tuple in the streamed change like we do for the
> > pg_decode_change?
> >
>
> I think so. The patch shows the message in
> pg_decode_stream_message(), so why to prohibit showing tuple here?
>
> > 2. pg_logical_slot_get_changes_guts
> > It recreate the decoding slot [ctx =
> > CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> > to false, should we pass a parameter to
> > pg_logical_slot_get_changes_guts saying whether we want streamed results or not
> >
>
> CreateDecodingContext internally calls StartupDecodingContext which
> sets the value of streaming based on if the plugin has provided
> callbacks for streaming functions. Isn't that sufficient? Why do we
> need additional parameters here?
I don't think that if plugin provides streaming function then we
should stream. Like pgoutput plugin provides streaming function but
we only stream if streaming is on in create subscription command. So
I feel that should be true with any plugin.
>
> > 3.
> > + XLogRecPtr prev_lsn = InvalidXLogRecPtr;
> > ReorderBufferChange *change;
> > ReorderBufferChange *specinsert = NULL;
> >
> > @@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > Relation relation = NULL;
> > Oid reloid;
> >
> > + /*
> > + * Enforce correct ordering of changes, merged from multiple
> > + * subtransactions. The changes may have the same LSN due to
> > + * MULTI_INSERT xlog records.
> > + */
> > + if (prev_lsn != InvalidXLogRecPtr)
> > + Assert(prev_lsn <= change->lsn);
> > +
> > + prev_lsn = change->lsn;
> > I did not understand, how this change is relavent to this patch
> >
>
> This is just to ensure that changes are in LSN order. I think as we
> are merging the changes before commit for streaming, it is good to
> have such an Assertion for ReorderBufferStreamTXN. And, if we want
> to have it in ReorderBufferStreamTXN, then there is no harm in keeping
> it in ReorderBufferCommit() at least to keep the code consistent. Do
> you see any problem with this?
I am fine with this.
>
> > 4.
> > + /*
> > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > + * information about subtransactions, which could arrive after streaming start.
> > + */
> > + if (!txn->is_schema_sent)
> > + snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > + txn, command_id);
> >
> > In which case, txn->is_schema_sent will be true, because at the end of
> > the stream in ReorderBufferExecuteInvalidations we are always setting
> > it false,
> > so while sending next stream it will always be false. That means we
> > never required snapshot_now variable in ReorderBufferTXN.
> >
>
> You are probably right, but as discussed we need to change this part
> of design/code (when to send schema changes) due to the issues
> discovered. So, I think this part will anyway change when we fix that
> problem.
Make sense.
>
> > 5.
> > @@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
> > *rb, TransactionId xid,
> > txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
> >
> > txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
> > +
> > + /*
> > + * We read catalog changes from WAL, which are not yet sent, so
> > + * invalidate current schema in order output plugin can resend
> > + * schema again.
> > + */
> > + txn->is_schema_sent = false;
> >
> > Same as point 4, during decode time it will never be true.
> >
>
> Sure, my previous point's reply applies here as well.
ok
>
> > 6.
> > + /* send fields */
> > + pq_sendint64(out, commit_lsn);
> > + pq_sendint64(out, txn->end_lsn);
> > + pq_sendint64(out, txn->commit_time);
> >
> > Commit_time and end_lsn is used in standby_feedback
> >
>
> I don't understand what you mean by this. Can you be a bit more clear?
I think I paste it here by mistake. just ignore it.
>
> >
> > 7.
> > + /* FIXME optimize the search by bsearch on sorted data */
> > + for (i = nsubxacts; i > 0; i--)
> > + {
> > + if (subxacts[i - 1].xid == subxid)
> > + {
> > + subidx = (i - 1);
> > + found = true;
> > + break;
> > + }
> > + }
> > We can not rollback intermediate subtransaction without rollbacking
> > latest sub-transaction, so why do we need
> > to search in the array? It will always be the the last subxact no?
> >
>
> The same thing is already mentioned in the comments above this code
> ("XXX Or perhaps we can rely on the aborts to arrive in the reverse
> order, i.e. from the inner-most subxact (when nested)? In which case
> we could simply check the last element."). I think what you are
> saying is probably right, but we can leave this as it is for now
> because this is a minor optimization which can be done later as well
> if required. However, if you see any correctness issue, then we can
> discuss.
I think more than optimization here we have the question of whether
this loop is required at all or not. Because, by optimizing we are
not adding the complexity, infact it will be simple. I think here we
need more analysis that whether we need to traverse the array or not.
So maybe for time being we can leave this as it is.
>
> > 8.
> > + /*
> > + * send feedback to upstream
> > + *
> > + * XXX Probably should send a valid LSN. But which one?
> > + */
> > + send_feedback(InvalidXLogRecPtr, false, false);
> >
> > Why feedback is sent for every change?
> >
>
> I will study this part of the patch and let you know my opinion.
Sure.
>
> Few comments on this patch series:
>
> 0001-Immediately-WAL-log-assignments:
> ------------------------------------------------------------
>
> The commit message still refers to the old design for this patch. I
> think you need to modify the commit message as per the latest patch.
>
> 0002-Issue-individual-invalidations-with-wal_level-log
> ----------------------------------------------------------------------------
> 1.
> xact_desc_invalidations(StringInfo buf,
> {
> ..
> + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
>
> You have removed logging for the above cache but forgot to remove its
> reference from one of the places. Also, I think you need to add a
> comment somewhere in inval.c to say why you are writing for WAL for
> some types of invalidations and not for others?
>
> 0003-Extend-the-output-plugin-API-with-stream-methods
> --------------------------------------------------------------------------------
> 1.
> + are required, while <function>stream_message_cb</function> and
> + <function>stream_message_cb</function> are optional.
>
> stream_message_cb is mentioned twice. It seems the second one is for truncate.
>
> 2.
> size of the transaction size and network bandwidth, the transfer time
> + may significantly increase the apply lag.
>
> /size of the transaction size/size of the transaction
>
> no need to mention size twice.
>
> 3.
> + Similarly to spill-to-disk behavior, streaming is triggered when the total
> + amount of changes decoded from the WAL (for all in-progress
> transactions)
> + exceeds limit defined by <varname>logical_work_mem</varname> setting.
>
> The guc name used is wrong. /Similarly to/Similar to/
>
> 4.
> stream_start_cb_wrapper()
> {
> ..
> + /* state.report_location = apply_lsn; */
> ..
> + /* FIXME ctx->write_location = apply_lsn; */
> ..
> }
>
> See, if we can fix these and similar in the callback for the stop. I
> think we don't have final_lsn till we commit/abort. Can we compute
> before calling these API's?
>
>
> 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> ----------------------------------------------------------------------------------
> 1.
> @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> PG_CATCH();
> {
> /* TODO: Encapsulate cleanup
> from the PG_TRY and PG_CATCH blocks */
> +
> if (iterstate)
> ReorderBufferIterTXNFinish(rb, iterstate);
>
> Spurious line change.
>
> 2. The commit message of this patch refers to Prepared transactions.
> I think that needs to be changed.
>
> 0006-Implement-streaming-mode-in-ReorderBuffer
> -------------------------------------------------------------------------
> 1.
> +
> +/* iterator for streaming (only get data from memory) */
> +static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
> +
> ReorderBuffer *rb,
> +
> ReorderBufferTXN
> *txn);
> +
> +static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
> + ReorderBuffer *rb,
> +
> ReorderBufferStreamIterTXNState * state);
> +
> +static void ReorderBufferStreamIterTXNFinish(
> +
> ReorderBuffer *rb,
> +
> ReorderBufferStreamIterTXNState * state);
>
> Do we really need to introduce new APIs for iterating over changes
> from streamed transactions? Why can't we reuse the same API's as we
> use for committed xacts?
>
> 2.
> +static void
> +ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
>
> Please write some comments atop ReorderBufferStreamCommit.
>
> 3.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> ..
> + if (txn->snapshot_now
> == NULL)
> + {
> + dlist_iter subxact_i;
> +
> + /* make sure this transaction is streamed for the first time */
> +
> Assert(!rbtxn_is_streamed(txn));
> +
> + /* at the beginning we should have invalid command ID */
> + Assert(txn->command_id ==
> InvalidCommandId);
> +
> + dlist_foreach(subxact_i, &txn->subtxns)
> + {
> + ReorderBufferTXN *subtxn;
> +
> +
> subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
> +
> + if (subtxn->base_snapshot != NULL &&
> +
> (txn->base_snapshot == NULL ||
> + txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
> + {
> +
> txn->base_snapshot = subtxn->base_snapshot;
>
> The logic here seems to be correct, but I am not sure why it is not
> considered to purge the base snapshot before assigning the subtxn's
> snapshot and similarly, we have not purged snapshot for subtxn once we
> are done with it. I think we can use
> ReorderBufferTransferSnapToParent to replace part of the logic here.
> Do you see any reason for doing things differently here?
>
> 4. In ReorderBufferStreamTXN, why do you need to use
> ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
>
> 5. I see a lot of code similarity in ReorderBufferStreamTXN and
> existing ReorderBufferCommit. I understand that there are some subtle
> differences due to which we need to write this new function but can't
> we encapsulate the specific parts of code in functions and then call
> from both places. I am talking about code in different cases for
> change->action.
>
> 6. + * Note: We never stream and serialize a transaction at the same time (e
> /(e/(we
>
I will look into these comments and reply separately.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-12 11:41 Amit Kapila <amit.kapila16@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-12 11:41 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Dec 11, 2019 at 11:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Dec 2, 2019 at 3:32 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > I have rebased the patch set on the latest head.
>
> 0001 looks like a clever approach, but are you sure it doesn't hurt
> performance when many small XLOG records are being inserted? I think
> XLogRecordAssemble() can get pretty hot in some workloads.
>
I don't think we have evaluated it yet, but we should do it. The
point to note is that it is only for the case when wal_level is
'logical' (see IsSubTransactionAssignmentPending) in which case we
already log more WAL, so this might not impact much. I guess that it
might be better to have that check in XLogRecordAssemble for the sake
of clarity.
>
> Regarding 0005, it seems to me that this is no good:
>
> + errmsg("improper heap_getnext call")));
>
> I think we should be using elog() rather than ereport() here, because
> this should only happen if there's a bug in a logical decoding plugin.
> At first, I thought maybe this should just be an Assert(), but since
> there are third-party logical decoding plugins available, checking
> this even in non-assert builds seems like a good idea. However, I
> think making it translatable is overkill; users should never see this,
> only developers.
>
makes sense. I think we should change it.
>
> + if (prev_lsn != InvalidXLogRecPtr)
> + Assert(prev_lsn <= change->lsn);
>
> There is no reason to ever write an if statement that contains only an
> Assert, and it's bad style. Write Assert(prev_lsn == InvalidXLogRecPtr
> || prev_lsn <= change->lsn), or better yet, use XLogRecPtrIsInvalid.
>
Agreed.
> The purpose and mechanism of the is_schema_sent flag is not clear to
> me. The word "schema" here seems to be being used to mean "snapshot,"
> which is rather confusing.
>
I have explained this flag below along with invalidations as both are
slightly related.
> I'm also somewhat unclear on what's happening here with invalidations.
> Perhaps that's as much a defect in my understanding as it is
> reflective of any problem with the patch, but I also don't see any
> comments either in 0002 or later patches explaining the theory of
> operation. If I've missed some, please point me in the right
> direction. Hypothetically speaking, it seems to me that if you just
> did InvalidateSystemCaches() every time the snapshot changed, you
> wouldn't need anything else (unless we're concerned with
> non-transactional invalidation messages like smgr and relmapper
> invalidations; not quite sure how those are handled). And, on the
> other hand, if we don't do InvalidateSystemCaches() every time the
> snapshot changes, then I don't understand why this works now, even
> without streaming.
>
I think the way invalidations work for logical replication is that
normally, we always start a new transaction before decoding each
commit which allows us to accept the invalidations (via
AtStart_Cache). However, if there are catalog changes within the
transaction being decoded, we need to reflect those before trying to
decode the WAL of operation which happened after that catalog change.
As we are not logging the WAL for each invalidation, we need to
execute all the invalidation messages for this transaction at each
catalog change. We are able to do that now as we decode the entire WAL
for a transaction only once we get the commit's WAL which contains all
the invalidation messages. So, we queue them up and execute them for
each catalog change which we identify by WAL record
XLOG_HEAP2_NEW_CID.
The second related concept is that before sending each change to
downstream (via pgoutput), we check whether we need to send the
schema. This we decide based on the local map entry
(RelationSyncEntry) which indicates whether the schema for the
relation is already sent or not. Once the schema of the relation is
sent, the entry for that relation in the map will indicate it. At the
time of invalidation processing we also blew up this map, so it always
reflects the correct state.
Now, to decode an in-progress transaction, we need to ensure that we
have received the WAL for all the invalidations before decoding the
WAL of action that happened immediately after that catalog change.
This is the reason we started WAL logging individual Invalidations.
So, with this change we don't need to execute all the invalidations
for each catalog change, rather execute them as and when their WAL is
being decoded.
The current mechanism to send schema changes won't work for streaming
transactions because after sending the change, subtransaction might
abort. On subtransaction abort, the downstream will simply discard
the changes where we will lose the previous schema change sent. There
is no such problem currently because we process all the aborts before
sending any change. So, the current idea of having a schema_sent flag
in each map entry (RelationSyncEntry) won't work for streaming
transactions. To solve this problem initially patch has kept a flag
'is_schema_sent' for each top-level transaction (in ReorderBufferTXN)
so that we can always send a schema for each (sub)transaction for
streaming transactions, but that won't work if we access multiple
relations in the same subtransaction. To solve this problem, we are
thinking of keeping a list/array of top-level xids in each
RelationSyncEntry. Basically, whenever we send the schema for any
transaction, we note that in RelationSyncEntry and at abort/commit
time we can remove xid from the list. Now, whenever, we check whether
to send schema for any operation in a transaction, we will check if
our xid is present in that list for a particular RelationSyncEntry and
take an action based on that (if xid is present, then we won't send
the schema, otherwise, send it). I think during decode, we should not
have that may open transactions, so the search in the array should be
cheap enough but we can consider some other data structure like hash
as well.
I will think some more and respond to your remaining comments/suggestions.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-13 09:16 Amit Kapila <amit.kapila16@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-13 09:16 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Dec 11, 2019 at 11:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Dec 2, 2019 at 3:32 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > I have rebased the patch set on the latest head.
>
> 0001 looks like a clever approach, but are you sure it doesn't hurt
> performance when many small XLOG records are being inserted? I think
> XLogRecordAssemble() can get pretty hot in some workloads.
>
> With regard to 0002, logging a separate WAL record for each
> invalidation seems painful; I think most operations that generate
> invalidations generate a bunch of them all at once. Perhaps you could
> just queue up invalidations as they happen, and then force anything
> that's been queued up to be emitted into WAL just before you emit any
> WAL record that might need to be decoded.
>
I feel we can log the invalidations of the entire command at one go if
we log at CommandEndInvalidationMessages. We already have all the
invalidations of current command in
transInvalInfo->CurrentCmdInvalidMsgs. This can save us the effort of
maintaining a new separate list/queue for invalidations and to a good
extent, it will ameliorate your concern of logging each invalidation
separately.
>
> 0006 contains lots of XXX comments that look like real issues. I guess
> those need to be fixed. Also, why don't we do the thing that the
> commit message for 0006 says we could "theoretically" do? I don't
> understand why we need the k-way merge at all,
>
I think we can do what is written in the commit message, but then we
need to maintain two paths (one for streaming contexts and other for
non-streaming contexts) unless we want to entirely get rid of storing
subtransaction changes separately which seems like a more fundamental
change. Right now, also to some extent such things are there, but I
have already given a comment to minimize it. Having said that, I
think we can go either way. I think the original intention was to
avoid doing more stuff unless it is really required as this is already
a big patchset, but maybe Tomas has a different idea about this.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-13 10:05 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-13 10:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 12, 2019 at 9:45 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have review the patch set and here are few comments/questions
> > >
> > > 1.
> > > +static void
> > > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > > + ReorderBufferTXN *txn,
> > > + Relation relation,
> > > + ReorderBufferChange *change)
> > > +{
> > > + OutputPluginPrepareWrite(ctx, true);
> > > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > > + OutputPluginWrite(ctx, true);
> > > +}
> > >
> > > Should we show the tuple in the streamed change like we do for the
> > > pg_decode_change?
> > >
> >
> > I think so. The patch shows the message in
> > pg_decode_stream_message(), so why to prohibit showing tuple here?
> >
> > > 2. pg_logical_slot_get_changes_guts
> > > It recreate the decoding slot [ctx =
> > > CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> > > to false, should we pass a parameter to
> > > pg_logical_slot_get_changes_guts saying whether we want streamed results or not
> > >
> >
> > CreateDecodingContext internally calls StartupDecodingContext which
> > sets the value of streaming based on if the plugin has provided
> > callbacks for streaming functions. Isn't that sufficient? Why do we
> > need additional parameters here?
>
> I don't think that if plugin provides streaming function then we
> should stream. Like pgoutput plugin provides streaming function but
> we only stream if streaming is on in create subscription command. So
> I feel that should be true with any plugin.
>
How about adding a new boolean parameter (streaming) in
pg_create_logical_replication_slot()?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-20 06:17 Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
4 siblings, 1 reply; 536+ messages in thread
From: Masahiko Sawada @ 2019-12-20 06:17 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, 2 Dec 2019 at 17:32, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > I have rebased the patch on the latest head and also fix the issue of
> > > "concurrent abort handling of the (sub)transaction." and attached as
> > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > the complete patch set. I have added the version number so that we
> > > can track the changes.
> >
> > The patch has rotten a bit and does not apply anymore. Could you
> > please send a rebased version? I have moved it to next CF, waiting on
> > author.
>
> I have rebased the patch set on the latest head.
Thank you for working on this.
This might have already been discussed but I have a question about the
changes of logical replication worker. In the current logical
replication there is a problem that the response time are doubled when
using synchronous replication because wal senders send changes after
commit. It's worse especially when a transaction makes a lot of
changes. So I expected this feature to reduce the response time by
sending changes even while the transaction is progressing but it
doesn't seem to be. The logical replication worker writes changes to
temporary files and applies these changes when the worker received
commit record (STREAM COMMIT). Since the worker sends the LSN of
commit record as flush LSN to the publisher after applying all
changes, the publisher must wait for all changes are applied to the
subscriber. Another problem would be that the worker doesn't receive
changes during applying changes of other transactions. These things
make me think it's better to have a new worker dedicated to apply
changes like we have the wal receiver process and the startup process.
Maybe we can have 2 workers (receiver and applyer) per subscriptions.
Any thoughts?
Regards,
--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-20 08:29 Kyotaro Horiguchi <horikyota.ntt@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kyotaro Horiguchi @ 2019-12-20 08:29 UTC (permalink / raw)
To: amit.kapila16@gmail.com; +Cc: robertmhaas@gmail.com; dilipbalaut@gmail.com; michael@paquier.xyz; tomas.vondra@2ndquadrant.com; peter.eisentraut@2ndquadrant.com; pgsql-hackers
Hello.
At Fri, 13 Dec 2019 14:46:20 +0530, Amit Kapila <amit.kapila16@gmail.com> wrote in
> On Wed, Dec 11, 2019 at 11:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Mon, Dec 2, 2019 at 3:32 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > I have rebased the patch set on the latest head.
> >
> > 0001 looks like a clever approach, but are you sure it doesn't hurt
> > performance when many small XLOG records are being inserted? I think
> > XLogRecordAssemble() can get pretty hot in some workloads.
> >
> > With regard to 0002, logging a separate WAL record for each
> > invalidation seems painful; I think most operations that generate
> > invalidations generate a bunch of them all at once. Perhaps you could
> > just queue up invalidations as they happen, and then force anything
> > that's been queued up to be emitted into WAL just before you emit any
> > WAL record that might need to be decoded.
> >
>
> I feel we can log the invalidations of the entire command at one go if
> we log at CommandEndInvalidationMessages. We already have all the
> invalidations of current command in
> transInvalInfo->CurrentCmdInvalidMsgs. This can save us the effort of
> maintaining a new separate list/queue for invalidations and to a good
> extent, it will ameliorate your concern of logging each invalidation
> separately.
I have a question on this. Does that mean that the current logical
decoder (or reorderbuffer) may emit incorrect result if it made a
catalog change during the current transaction being decoded? If so,
this is not a feature but a bug fix.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-20 13:30 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-20 13:30 UTC (permalink / raw)
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Dec 20, 2019 at 11:47 AM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Mon, 2 Dec 2019 at 17:32, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> > >
> > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > > I have rebased the patch on the latest head and also fix the issue of
> > > > "concurrent abort handling of the (sub)transaction." and attached as
> > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > > the complete patch set. I have added the version number so that we
> > > > can track the changes.
> > >
> > > The patch has rotten a bit and does not apply anymore. Could you
> > > please send a rebased version? I have moved it to next CF, waiting on
> > > author.
> >
> > I have rebased the patch set on the latest head.
>
> Thank you for working on this.
>
> This might have already been discussed but I have a question about the
> changes of logical replication worker. In the current logical
> replication there is a problem that the response time are doubled when
> using synchronous replication because wal senders send changes after
> commit. It's worse especially when a transaction makes a lot of
> changes. So I expected this feature to reduce the response time by
> sending changes even while the transaction is progressing but it
> doesn't seem to be. The logical replication worker writes changes to
> temporary files and applies these changes when the worker received
> commit record (STREAM COMMIT). Since the worker sends the LSN of
> commit record as flush LSN to the publisher after applying all
> changes, the publisher must wait for all changes are applied to the
> subscriber.
>
The main aim of this feature is to reduce apply lag. Because if we
send all the changes together it can delay there apply because of
network delay, whereas if most of the changes are already sent, then
we will save the effort on sending the entire data at commit time.
This in itself gives us decent benefits. Sure, we can further improve
it by having separate workers (dedicated to apply the changes) as you
are suggesting and in fact, there is a patch for that as well(see the
performance results and bgworker patch at [1]), but if try to shove in
all the things in one go, then it will be difficult to get this patch
committed (there are already enough things and the patch is quite big
that to get it right takes a lot of energy). So, the plan is
something like that first we get the basic feature and then try to
improve by having dedicated workers or things like that. Does this
make sense to you?
[1] - https://www.postgresql.org/message-id/8eda5118-2dd0-79a1-4fe9-eec7e334de17%40postgrespro.ru
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-20 13:42 Amit Kapila <amit.kapila16@gmail.com>
parent: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-20 13:42 UTC (permalink / raw)
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Dec 20, 2019 at 2:00 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
>
> Hello.
>
> At Fri, 13 Dec 2019 14:46:20 +0530, Amit Kapila <amit.kapila16@gmail.com> wrote in
> > On Wed, Dec 11, 2019 at 11:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > On Mon, Dec 2, 2019 at 3:32 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > I have rebased the patch set on the latest head.
> > >
> > > 0001 looks like a clever approach, but are you sure it doesn't hurt
> > > performance when many small XLOG records are being inserted? I think
> > > XLogRecordAssemble() can get pretty hot in some workloads.
> > >
> > > With regard to 0002, logging a separate WAL record for each
> > > invalidation seems painful; I think most operations that generate
> > > invalidations generate a bunch of them all at once. Perhaps you could
> > > just queue up invalidations as they happen, and then force anything
> > > that's been queued up to be emitted into WAL just before you emit any
> > > WAL record that might need to be decoded.
> > >
> >
> > I feel we can log the invalidations of the entire command at one go if
> > we log at CommandEndInvalidationMessages. We already have all the
> > invalidations of current command in
> > transInvalInfo->CurrentCmdInvalidMsgs. This can save us the effort of
> > maintaining a new separate list/queue for invalidations and to a good
> > extent, it will ameliorate your concern of logging each invalidation
> > separately.
>
> I have a question on this. Does that mean that the current logical
> decoder (or reorderbuffer)
>
What does currently refer to here? Is it about HEAD or about the
patch? Without the patch, we decode only at commit time and by that
time we have all invalidations (logged with commit WAL record), so we
just execute them at each catalog change (see the actions in
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID). The patch has to
separately WAL log each invalidation because we can decode the
intermittent changes, so we can't wait till commit. The above is just
an optimization for the patch. AFAIK, there is no correctness issue
here, but let me know if you see any.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-22 11:34 vignesh C <vignesh21@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
4 siblings, 1 reply; 536+ messages in thread
From: vignesh C @ 2019-12-22 11:34 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > I have rebased the patch on the latest head and also fix the issue of
> > > "concurrent abort handling of the (sub)transaction." and attached as
> > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > the complete patch set. I have added the version number so that we
> > > can track the changes.
> >
> > The patch has rotten a bit and does not apply anymore. Could you
> > please send a rebased version? I have moved it to next CF, waiting on
> > author.
>
> I have rebased the patch set on the latest head.
>
Few comments:
assert variable should be within #ifdef USE_ASSERT_CHECKING in patch
v2-0008-Add-support-for-streaming-to-built-in-replication.patch:
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
Add the typedefs like below in typedefs.lst common across the patches:
xl_xact_invalidations, ReorderBufferStreamIterTXNEntry,
ReorderBufferStreamIterTXNState, SubXactInfo
"are written" appears twice in commit message of
v2-0002-Issue-individual-invalidations-with-wal_level-log.patch:
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
v2-0002-Issue-individual-invalidations-with-wal_level-log.patch patch
does not compile by itself:
reorderbuffer.c:1822:9: error: ‘ReorderBufferTXN’ has no member named
‘is_schema_sent’
+
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
Should we include printing of id here like in earlier cases in
v2-0002-Issue-individual-invalidations-with-wal_level-log.patch:
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
There is some code duplication in stream_change_cb_wrapper,
stream_truncate_cb_wrapper, stream_message_cb_wrapper,
stream_abort_cb_wrapper, stream_commit_cb_wrapper,
stream_start_cb_wrapper and stream_stop_cb_wrapper functions in
v2-0003-Extend-the-output-plugin-API-with-stream-methods.patch patch.
Should we have a separate function for common code?
Should we can add function header for AssertChangeLsnOrder in
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch:
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
This "Assert(txn->first_lsn != InvalidXLogRecPtr)"can be before the
loop, can be checked only once:
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange,
node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
Should we add function header for ReorderBufferDestroyTupleCidHash in
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch:
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
Should we add function header for ReorderBufferStreamCommit in
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch:
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
Should we add function header for ReorderBufferCanStream in
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch:
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
patch v2-0008-Add-support-for-streaming-to-built-in-replication.patch
does not apply:
Hunk #18 FAILED at 2035.
Hunk #19 succeeded at 2199 (offset -16 lines).
1 out of 19 hunks FAILED -- saving rejects to file
src/backend/replication/logical/worker.c.rej
Header inclusion may not be required in patch
v2-0008-Add-support-for-streaming-to-built-in-replication.patch:
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-24 04:24 Amit Kapila <amit.kapila16@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-24 04:24 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Dec 22, 2019 at 5:04 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Few comments:
> assert variable should be within #ifdef USE_ASSERT_CHECKING in patch
> v2-0008-Add-support-for-streaming-to-built-in-replication.patch:
> + int64 subidx;
> + bool found = false;
> + char path[MAXPGPATH];
> +
> + subidx = -1;
> + subxact_info_read(MyLogicalRepWorker->subid, xid);
> +
> + /* FIXME optimize the search by bsearch on sorted data */
> + for (i = nsubxacts; i > 0; i--)
> + {
> + if (subxacts[i - 1].xid == subxid)
> + {
> + subidx = (i - 1);
> + found = true;
> + break;
> + }
> + }
> +
> + /* We should not receive aborts for unknown subtransactions. */
> + Assert(found);
>
We can use PG_USED_FOR_ASSERTS_ONLY for that variable.
>
> Should we include printing of id here like in earlier cases in
> v2-0002-Issue-individual-invalidations-with-wal_level-log.patch:
> + appendStringInfo(buf, " relcache %u", msg->rc.relId);
> + /* not expected, but print something anyway */
> + else if (msg->id == SHAREDINVALSMGR_ID)
> + appendStringInfoString(buf, " smgr");
> + /* not expected, but print something anyway */
> + else if (msg->id == SHAREDINVALRELMAP_ID)
> + appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
>
I am not sure if this patch is logging these invalidations, so not
sure if it makes sense to add more ids in the cases you are referring
to. However, if we change it to logging all invalidations at command
end as being discussed in this thread, then it might be better to do
what you are suggesting.
>
> Should we can add function header for AssertChangeLsnOrder in
> v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch:
> +static void
> +AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
> +{
>
> This "Assert(txn->first_lsn != InvalidXLogRecPtr)"can be before the
> loop, can be checked only once:
> + dlist_foreach(iter, &txn->changes)
> + {
> + ReorderBufferChange *cur_change;
> +
> + cur_change = dlist_container(ReorderBufferChange,
> node, iter.cur);
> +
> + Assert(txn->first_lsn != InvalidXLogRecPtr);
> + Assert(cur_change->lsn != InvalidXLogRecPtr);
> + Assert(txn->first_lsn <= cur_change->lsn);
>
This makes sense to me. Another thing about this function, do we
really need "ReorderBuffer *rb" parameter in this function?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-24 05:28 Robert Haas <robertmhaas@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Robert Haas @ 2019-12-24 05:28 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 12, 2019 at 3:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> I don't think we have evaluated it yet, but we should do it. The
> point to note is that it is only for the case when wal_level is
> 'logical' (see IsSubTransactionAssignmentPending) in which case we
> already log more WAL, so this might not impact much. I guess that it
> might be better to have that check in XLogRecordAssemble for the sake
> of clarity.
I don't think that this is really a valid argument. Just because we
have some overhead now doesn't mean that adding more won't hurt. Even
testing the wal_level costs a little something.
> I think the way invalidations work for logical replication is that
> normally, we always start a new transaction before decoding each
> commit which allows us to accept the invalidations (via
> AtStart_Cache). However, if there are catalog changes within the
> transaction being decoded, we need to reflect those before trying to
> decode the WAL of operation which happened after that catalog change.
> As we are not logging the WAL for each invalidation, we need to
> execute all the invalidation messages for this transaction at each
> catalog change. We are able to do that now as we decode the entire WAL
> for a transaction only once we get the commit's WAL which contains all
> the invalidation messages. So, we queue them up and execute them for
> each catalog change which we identify by WAL record
> XLOG_HEAP2_NEW_CID.
Thanks for the explanation. That makes sense. But, it's still true,
AFAICS, that instead of doing this stuff with logging invalidations
you could just InvalidateSystemCaches() in the cases where you are
currently applying all of the transaction's invalidations. That
approach might be worse than changing the way invalidations are
logged, but the two approaches deserve to be compared. One approach
has more CPU overhead and the other has more WAL overhead, so it's a
little hard to compare them, but it seems worth mulling over.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-24 05:46 Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Masahiko Sawada @ 2019-12-24 05:46 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, 20 Dec 2019 at 22:30, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Dec 20, 2019 at 11:47 AM Masahiko Sawada
> <masahiko.sawada@2ndquadrant.com> wrote:
> >
> > On Mon, 2 Dec 2019 at 17:32, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> > > >
> > > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > > > I have rebased the patch on the latest head and also fix the issue of
> > > > > "concurrent abort handling of the (sub)transaction." and attached as
> > > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > > > the complete patch set. I have added the version number so that we
> > > > > can track the changes.
> > > >
> > > > The patch has rotten a bit and does not apply anymore. Could you
> > > > please send a rebased version? I have moved it to next CF, waiting on
> > > > author.
> > >
> > > I have rebased the patch set on the latest head.
> >
> > Thank you for working on this.
> >
> > This might have already been discussed but I have a question about the
> > changes of logical replication worker. In the current logical
> > replication there is a problem that the response time are doubled when
> > using synchronous replication because wal senders send changes after
> > commit. It's worse especially when a transaction makes a lot of
> > changes. So I expected this feature to reduce the response time by
> > sending changes even while the transaction is progressing but it
> > doesn't seem to be. The logical replication worker writes changes to
> > temporary files and applies these changes when the worker received
> > commit record (STREAM COMMIT). Since the worker sends the LSN of
> > commit record as flush LSN to the publisher after applying all
> > changes, the publisher must wait for all changes are applied to the
> > subscriber.
> >
>
> The main aim of this feature is to reduce apply lag. Because if we
> send all the changes together it can delay there apply because of
> network delay, whereas if most of the changes are already sent, then
> we will save the effort on sending the entire data at commit time.
> This in itself gives us decent benefits. Sure, we can further improve
> it by having separate workers (dedicated to apply the changes) as you
> are suggesting and in fact, there is a patch for that as well(see the
> performance results and bgworker patch at [1]), but if try to shove in
> all the things in one go, then it will be difficult to get this patch
> committed (there are already enough things and the patch is quite big
> that to get it right takes a lot of energy). So, the plan is
> something like that first we get the basic feature and then try to
> improve by having dedicated workers or things like that. Does this
> make sense to you?
>
Thank you for explanation. The plan makes sense. But I think in the
current design it's a problem that logical replication worker doesn't
receive changes (and doesn't check interrupts) during applying
committed changes even if we don't have a worker dedicated for
applying. I think the worker should continue to receive changes and
save them to temporary files even during applying changes. Otherwise
the buffer would be easily full and replication gets stuck.
Regards,
--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-24 08:21 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-24 08:21 UTC (permalink / raw)
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Dec 24, 2019 at 11:17 AM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Fri, 20 Dec 2019 at 22:30, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > The main aim of this feature is to reduce apply lag. Because if we
> > send all the changes together it can delay there apply because of
> > network delay, whereas if most of the changes are already sent, then
> > we will save the effort on sending the entire data at commit time.
> > This in itself gives us decent benefits. Sure, we can further improve
> > it by having separate workers (dedicated to apply the changes) as you
> > are suggesting and in fact, there is a patch for that as well(see the
> > performance results and bgworker patch at [1]), but if try to shove in
> > all the things in one go, then it will be difficult to get this patch
> > committed (there are already enough things and the patch is quite big
> > that to get it right takes a lot of energy). So, the plan is
> > something like that first we get the basic feature and then try to
> > improve by having dedicated workers or things like that. Does this
> > make sense to you?
> >
>
> Thank you for explanation. The plan makes sense. But I think in the
> current design it's a problem that logical replication worker doesn't
> receive changes (and doesn't check interrupts) during applying
> committed changes even if we don't have a worker dedicated for
> applying. I think the worker should continue to receive changes and
> save them to temporary files even during applying changes.
>
Won't it beat the purpose of this feature which is to reduce the apply
lag? Basically, it can so happen that while applying commit, it
constantly gets changes of other transactions which will delay the
apply of the current transaction. Also, won't it create some further
work to identify the order of commits? Say while applying commit-1,
it receives 5 other commits that are written to separate temporary
files. How will we later identify which transaction's WAL we need to
apply first? We might deduce by LSN's, but I think that could be
tricky. Another thing is that I think it could lead to some design
complications as well because while applying commit, you need some
sort of callback or something like that to receive and flush totally
unrelated changes. It could lead to another kind of failure mode
wherein while applying commit if it tries to receive another
transaction data and some failure happens while writing the data of
that transaction. I am not sure if it is a good idea to try something
like that.
> Otherwise
> the buffer would be easily full and replication gets stuck.
>
Are you telling about network buffer? I think the best way as
discussed is to launch new workers for streamed transactions, but we
can do that as an additional feature. Anyway, as proposed, users can
choose the streaming mode for subscriptions, so there is an option to
turn this selectively.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-26 07:05 Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Masahiko Sawada @ 2019-12-26 07:05 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, 24 Dec 2019 at 17:21, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Dec 24, 2019 at 11:17 AM Masahiko Sawada
> <masahiko.sawada@2ndquadrant.com> wrote:
> >
> > On Fri, 20 Dec 2019 at 22:30, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > The main aim of this feature is to reduce apply lag. Because if we
> > > send all the changes together it can delay there apply because of
> > > network delay, whereas if most of the changes are already sent, then
> > > we will save the effort on sending the entire data at commit time.
> > > This in itself gives us decent benefits. Sure, we can further improve
> > > it by having separate workers (dedicated to apply the changes) as you
> > > are suggesting and in fact, there is a patch for that as well(see the
> > > performance results and bgworker patch at [1]), but if try to shove in
> > > all the things in one go, then it will be difficult to get this patch
> > > committed (there are already enough things and the patch is quite big
> > > that to get it right takes a lot of energy). So, the plan is
> > > something like that first we get the basic feature and then try to
> > > improve by having dedicated workers or things like that. Does this
> > > make sense to you?
> > >
> >
> > Thank you for explanation. The plan makes sense. But I think in the
> > current design it's a problem that logical replication worker doesn't
> > receive changes (and doesn't check interrupts) during applying
> > committed changes even if we don't have a worker dedicated for
> > applying. I think the worker should continue to receive changes and
> > save them to temporary files even during applying changes.
> >
>
> Won't it beat the purpose of this feature which is to reduce the apply
> lag? Basically, it can so happen that while applying commit, it
> constantly gets changes of other transactions which will delay the
> apply of the current transaction.
You're right. But it seems to me that it optimizes the apply lags of
only a transaction that made many changes. On the other hand if a
transaction made many changes applying of subsequent changes are
delayed.
> Also, won't it create some further
> work to identify the order of commits? Say while applying commit-1,
> it receives 5 other commits that are written to separate temporary
> files. How will we later identify which transaction's WAL we need to
> apply first? We might deduce by LSN's, but I think that could be
> tricky. Another thing is that I think it could lead to some design
> complications as well because while applying commit, you need some
> sort of callback or something like that to receive and flush totally
> unrelated changes. It could lead to another kind of failure mode
> wherein while applying commit if it tries to receive another
> transaction data and some failure happens while writing the data of
> that transaction. I am not sure if it is a good idea to try something
> like that.
It's just an idea but we might want to have new workers dedicated to
apply changes first and then we will have streaming option later. That
way we can reduce the flush lags depending on use cases. The commit
order can be determined by the receiver and shared with the applyer
in shared memory. Once we separated workers the streaming option can
be introduced without such a downside.
>
> > Otherwise
> > the buffer would be easily full and replication gets stuck.
> >
>
> Are you telling about network buffer?
Yes.
> I think the best way as
> discussed is to launch new workers for streamed transactions, but we
> can do that as an additional feature. Anyway, as proposed, users can
> choose the streaming mode for subscriptions, so there is an option to
> turn this selectively.
Yes. But user who wants to use this feature would want to replicate
many changes but I guess the side effect is quite big. I think that at
least we need to make the logical replication tolerate such situation.
Regards,
--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-28 16:03 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-12-28 16:03 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Dec 10, 2019 at 10:23:19AM +0530, Dilip Kumar wrote:
>On Tue, Dec 10, 2019 at 9:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
>> > >
>> > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
>> > > > I have rebased the patch on the latest head and also fix the issue of
>> > > > "concurrent abort handling of the (sub)transaction." and attached as
>> > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
>> > > > the complete patch set. I have added the version number so that we
>> > > > can track the changes.
>> > >
>> > > The patch has rotten a bit and does not apply anymore. Could you
>> > > please send a rebased version? I have moved it to next CF, waiting on
>> > > author.
>> >
>> > I have rebased the patch set on the latest head.
>> >
>> > Apart from this, there is one issue reported by my colleague Vignesh.
>> > The issue is that if we use more than two relations in a transaction
>> > then there is an error on standby (no relation map entry for remote
>> > relation ID 16390). After analyzing I have found that for the
>> > streaming transaction an "is_schema_sent" flag is kept in
>> > ReorderBufferTXN. And, I think that is done so that we can send the
>> > schema for each transaction stream so that if any subtransaction gets
>> > aborted we don't lose the logical WAL for that schema. But, this
>> > solution has induced a very basic issue that if a transaction operate
>> > on more than 1 relation then after sending the schema for the first
>> > relation it will mark the flag true and the schema for the subsequent
>> > relations will never be sent.
>> >
>>
>> How about keeping a list of top-level xids in each RelationSyncEntry?
>> Basically, whenever we send the schema for any transaction, we note
>> that in RelationSyncEntry and at abort time we can remove xid from the
>> list. Now, whenever, we check whether to send schema for any
>> operation in a transaction, we will check if our xid is present in
>> that list for a particular RelationSyncEntry and take an action based
>> on that (if xid is present, then we won't send the schema, otherwise,
>> send it).
>The idea make sense to me. I will try to write a patch for this and test.
>
Yeah, the "is_schema_sent" flag in ReorderBufferTXN does not work - it
needs to be in the RelationSyncEntry. In fact, I already have code for
that in my private repository - I thought the patches I sent here do
include this, but apparently I forgot to include this bit :-(
Attached is a rebased patch series, fixing this. It's essentially v2
with a couple of patches (0003, 0008, 0009 and 0012) replacing the
is_schema_sent with correct handling.
0003 - removes an is_schema_sent reference added prematurely (it's added
by a later patch, causing compile failure)
0008 - adds the is_schema_sent back (essentially reverting 0003)
0009 - removes is_schema_sent entirely
0012 - adds the correct handling of schema flags in pgoutput
I don't know what other changes you've made since v2, so this way it
should be possible to just take 0003, 0008, 0009 and 0012 and slip them
in with minimal hassle.
FWIW thanks to everyone (and Amit and Dilip in particular) working on
this patch series. There's been a lot of great reviews and improvements
since I abandoned this thread for a while. I expect to be able to spend
more time working on this in January.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From ae907baaaf7401a4ac906de6127f4318241ac3a5 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH 01/17] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++--
src/backend/access/transam/xlogreader.c | 5 +++
src/backend/replication/logical/decode.c | 39 ++++++++++----------
src/include/access/xact.h | 3 ++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 ++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6ab0b..708e5233f4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0036..a8a8084713 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b05f1..4435c636bc 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d027b..897b755eb4 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899dea1..5b9740c5c3 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3fea1993bc..b1976ac653 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611b7f..a676151561 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54195..bcfba0a101 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
2.21.0
From 7b2a948ef7ca6fa43f94ccf11f11f8edfb3fe028 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH 02/17] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 ++++++++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 23 +++++++
.../replication/logical/reorderbuffer.c | 56 +++++++++++++--
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++
src/include/access/xact.h | 18 ++++-
src/include/replication/reorderbuffer.h | 14 ++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5322..6cfd6af24e 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e5233f4..da15556357 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755eb4..9bcefb6e6d 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb877..b1feff3e71 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2207,6 +2214,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9aff..0682c55b51 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c5c3..82d49428c2 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -197,6 +197,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
+/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9e63..6a7187bbec 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
2.21.0
From 473be553d5d798eac72700a5352d099295344cd6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH 03/17] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3e71..c0b97251e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1819,7 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
2.21.0
From c6f6ee18707a3dea988b98c6c5351d816a67be70 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/17] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 ++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 +++++
src/include/replication/reorderbuffer.h | 57 ++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4bded..9c77791dd5 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db968641e..fc4ad65eae 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615864..b88b58505a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e6d2..1e934d25e6 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -79,6 +79,11 @@ typedef struct LogicalDecodingContext
*/
void *output_writer_private;
+ /*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
/*
* State for writing output.
*/
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f26d..a30546250a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -99,6 +99,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
+/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
/*
* Output plugin callbacks
*/
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187bbec..5b4be2bf88 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -383,6 +429,17 @@ struct ReorderBuffer
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
+ /*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
/*
* Pointer that will be passed untouched to the callbacks.
*/
--
2.21.0
From 8452286c8256ba59163f98309c7c59cefbe64845 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/17] Cleaning up of flags in ReorderBufferTXN structure
---
.../replication/logical/reorderbuffer.c | 36 +++++++++----------
src/include/replication/reorderbuffer.h | 33 ++++++++++-------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c0b97251e2..f74c1996d0 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1969,7 +1969,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2117,7 +2117,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2296,7 +2296,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2313,7 +2313,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2333,7 +2333,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2521,12 +2521,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2bf88..19c7bac8c0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -248,15 +264,6 @@ typedef struct ReorderBufferTXN
*/
uint64 nentries_mem;
- /*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
/*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
--
2.21.0
From 3656702c911785df27422ecd177e506723b0e33f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH 06/17] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 +-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++
.../replication/logical/reorderbuffer.c | 9 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65eae..da6a6f3233 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb34ef..2a60a7380a 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1421,6 +1432,16 @@ heap_fetch(Relation relation,
OffsetNumber offnum;
bool valid;
+ /*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
/*
* Fetch and pin the appropriate page of the relation.
*/
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1682,6 +1713,16 @@ heap_get_latest_tid(TableScanDesc sscan,
*/
Assert(ItemPointerIsValid(tid));
+ /*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
/*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d342..201acfbbf2 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f74c1996d0..76d2701233 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1876,6 +1876,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517596..9fa1e43347 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -153,6 +153,13 @@ static Snapshot SecondarySnapshot = NULL;
static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
+/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df48c..9a8f9ceba3 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
2.21.0
From 4dfb923fb4eea66cffb58de3175ae44fa56350b3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH 07/17] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
.../replication/logical/reorderbuffer.c | 1075 ++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e3646716f..cf10dd041d 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 76d2701233..f02c47238a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1234,6 +1319,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
pfree(state);
}
+/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1514,6 +1897,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_id = origin_id;
txn->origin_lsn = origin_lsn;
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
/*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1929,6 +2339,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2013,6 +2430,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2148,8 +2572,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2157,6 +2590,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2168,19 +2602,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2209,6 +2652,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2284,6 +2728,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2298,6 +2745,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2401,6 +2865,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
return largest;
}
+/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
@@ -2421,15 +2917,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2722,6 +3249,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac8c0..7d08e2fd39 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -221,6 +236,16 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
/*
* LSN pointing to the end of the commit record + 1.
*/
@@ -251,6 +276,13 @@ typedef struct ReorderBufferTXN
XLogRecPtr base_snapshot_lsn;
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
+ /*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
/*
* How many ReorderBufferChange's do we have in this txn.
*
--
2.21.0
From 6e512a0622a5a532aaa788b049ba028aa8fc0115 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:02:38 +0100
Subject: [PATCH 08/17] fixup: add is_schema_sent back
---
src/backend/replication/logical/reorderbuffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f02c47238a..0ab319182a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,6 +2229,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
2.21.0
From 2a7baf771f161a165907b7f8038aed6d12e3cc36 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:46:15 +0100
Subject: [PATCH 09/17] fixup: get rid of is_schema_sent entirely
We'll do this in the pgoutput.c code directly, not in reorderbuffer.
---
.../replication/logical/reorderbuffer.c | 26 ++-----------------
src/include/replication/reorderbuffer.h | 5 ----
2 files changed, 2 insertions(+), 29 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab319182a..85db15ea3b 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,7 +2229,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
@@ -2729,9 +2728,6 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
-
- /* Invalidate current schema as well */
- txn->is_schema_sent = false;
}
/*
@@ -2747,22 +2743,12 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- /*
- * We read catalog changes from WAL, which are not yet sent, so
- * invalidate current schema in order output plugin can resend
- * schema again.
- */
- txn->is_schema_sent = false;
-
/*
* TOCHECK: Mark toplevel transaction as having catalog changes too
* if one of its children has.
*/
if (txn->toptxn != NULL)
- {
txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- txn->toptxn->is_schema_sent = false;
- }
}
/*
@@ -3345,9 +3331,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* TOCHECK: We have to rebuild historic snapshot to be sure it includes all
* information about subtransactions, which could arrive after streaming start.
*/
- if (!txn->is_schema_sent)
- snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
}
/*
@@ -3602,12 +3587,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
snapshot_now = change->data.snapshot;
}
- /*
- * TOCHECK: Snapshot changed, then invalidate current schema to reflect
- * possible catalog changes.
- */
- txn->is_schema_sent = false;
-
/* and continue with the new one */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
txn->xid);
@@ -3646,7 +3625,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2fd39..e2b8db0ff1 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -236,11 +236,6 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
- /*
- * Do we need to send schema for this transaction in output plugin?
- */
- bool is_schema_sent;
-
/*
* Toplevel transaction for this subxact (NULL for top-level).
*/
--
2.21.0
From 129547a473edd42fa66f1490d36f8aaede529867 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH 10/17] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 ++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c90282f..8b1923c9de 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..91790b0c95 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff499..2a276482c1 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edcfc2..fbb447379f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fcd05..0ab6855ad8 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 63ba0ae234..c80acd3eb0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1729,6 +1729,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b877..cf6e03b9a7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..10ea113e4d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714eaf0c..1db706af54 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.21.0
From 83135d7f8db59c80c2ab3b1a5c5542d9b0400fdb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH 11/17] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 +++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../t/012_stream_subxact_abort.pl | 82 ++
.../t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..e1fb9075e1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0c95..d9abf5e64c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a276482c1..15a6f5a8b3 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb447379f..b2b93d6234 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7410b2ff5e..a479ce9329 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855ad8..9970170e47 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index c57b578b48..0a013ed220 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b58505a..ad43ab365e 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..5a379fb6bc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c80acd3eb0..cf053e948b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
errmsg("ORIGIN message sent out of order")));
}
+/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
/*
* Handle RELATION message.
*
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -694,6 +1122,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -814,6 +1245,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -913,6 +1347,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1004,6 +1441,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1100,6 +1553,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
}
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
/*
* Apply main loop.
*/
@@ -1116,6 +1585,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1564,6 +2036,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1730,6 +2760,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b9a7..8490ea4717 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -512,6 +638,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
rel_sync_cache_publication_cb(arg, cacheid, hashvalue);
}
+/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
/*
* Initialize the relation schema sync cache for a decoding session.
*
@@ -622,6 +833,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
/*
* Relcache invalidation callback
*/
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ba08ad405f..8eb3160041 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1f23665432..2e0743ac8f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndPrepareWrite, WalSndWriteData,
WalSndUpdateProgress);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
/*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113e4d..8793676258 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2e873d048..c522703d8c 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..bf02cbc19d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706af54..3d19b5d88e 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000000..2f01133f69
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000000..d2ae38592b
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000000..0da39a1a8a
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000000..402df30f59
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000000..becbdd0578
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.21.0
From 7a45713c4161e4efdf82148f56af3328283b20e9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:56:04 +0100
Subject: [PATCH 12/17] fixup: add proper schema tracking
---
src/backend/replication/pgoutput/pgoutput.c | 45 ++++++++++++++++++++-
1 file changed, 43 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 8490ea4717..0148f4c01e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -82,6 +82,8 @@ typedef struct RelationSyncEntry
Oid relid; /* relation oid */
TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -96,6 +98,11 @@ static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -366,6 +373,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
bool schema_sent;
TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
/*
* Remember XID of the (sub)transaction for the change. We don't care if
@@ -378,6 +386,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (in_streaming)
xid = change->txn->xid;
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
/*
* Do we need to send the schema? We do track streamed transactions
* separately, because those may not be applied later (and the regular
@@ -391,7 +404,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
* occur when streaming already started, so we have to track new catalog
* changes somehow.
*/
- schema_sent = txn->is_schema_sent;
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
}
else
schema_sent = relentry->schema_sent;
@@ -432,7 +445,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->xid = change->txn->xid;
if (in_streaming)
- txn->is_schema_sent = true;
+ set_schema_sent_in_streamed_txn(relentry, topxid);
else
relentry->schema_sent = true;
}
@@ -759,6 +772,34 @@ init_rel_sync_cache(MemoryContext cachectx)
(Datum) 0);
}
+/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
/*
* Find or create entry in the relation schema cache.
*/
--
2.21.0
From 0b9a94e41e81ef0ce58ada49da55cb10e2cc5a1f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH 13/17] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
.../replication/logical/reorderbuffer.c | 13 ++++++++
src/backend/replication/walsender.c | 32 ++++++++++++++++---
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 13 +++++---
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 ++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb58115af..180ea880a4 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..58976118db 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 85db15ea3b..65f876d0f0 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3709,6 +3713,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_END_TRY();
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
/*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2e0743ac8f..9f93f11d8a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3185,7 +3189,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3242,6 +3246,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3265,6 +3272,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3351,6 +3361,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3598,12 +3613,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b219..3b897a5d70 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e2b8db0ff1..e132c3c5ea 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -506,15 +506,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b32051ac..7efc332319 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a07825b9..5ab21a80d0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
2.21.0
From 3b857c0da517c7aed243253e8411767a0e973d63 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH 14/17] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560b23..8cd1993393 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2fbc..94c71f8ae2 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f65fa..8dfeafc772 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9181..a6fae9c3f1 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17a78..202871a658 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10a19..70c86b22ac 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6d63..f9c8d1d348 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7332..0c9c6b3dd4 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bdba9..21f50c7012 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133f69..30561d8f96 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae38592b..9a6bac6822 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bdc35..ed56fbf96c 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cba4c..4df1ddef63 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1a8a..c3caff6149 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7c2f..c62eb521e7 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30f59..2be7542831 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0578..2da9607a7d 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9435..96ffc091b0 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
2.21.0
From 016b2a43955bb56f3a439d7a003fe9d0d6412125 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 15/17] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 65f876d0f0..5883d14a02 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
2.21.0
From 9fd14e817ba569654dc32ef75aadde7300ad1b0f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 16/17] Add TAP test for streaming vs. DDL
---
.../subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000000..b8d78b1972
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.21.0
From c9d0c5da2641d0746657d465893bb65eb9d73b7e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH 17/17] Extend handling of concurrent aborts for streaming
transaction
---
.../replication/logical/reorderbuffer.c | 36 +++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 +++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5883d14a02..0987032208 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2349,9 +2349,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3267,6 +3267,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3396,6 +3397,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3457,6 +3465,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3695,6 +3707,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3713,7 +3728,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e132c3c5ea..6186465a85 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -236,6 +236,11 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
/*
* Toplevel transaction for this subxact (NULL for top-level).
*/
--
2.21.0
Attachments:
0001-Immediately-WAL-log-assignments-v3.patch (text/plain, 10.2K) download
From ae907baaaf7401a4ac906de6127f4318241ac3a5 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH 01/17] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++--
src/backend/access/transam/xlogreader.c | 5 +++
src/backend/replication/logical/decode.c | 39 ++++++++++----------
src/include/access/xact.h | 3 ++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 ++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6ab0b..708e5233f4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0036..a8a8084713 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b05f1..4435c636bc 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d027b..897b755eb4 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899dea1..5b9740c5c3 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3fea1993bc..b1976ac653 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611b7f..a676151561 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54195..bcfba0a101 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
2.21.0
0002-Issue-individual-invalidations-with-wal_level-log-v3.patch (text/plain, 15.7K) download
From 7b2a948ef7ca6fa43f94ccf11f11f8edfb3fe028 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH 02/17] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 ++++++++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 23 +++++++
.../replication/logical/reorderbuffer.c | 56 +++++++++++++--
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++
src/include/access/xact.h | 18 ++++-
src/include/replication/reorderbuffer.h | 14 ++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5322..6cfd6af24e 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e5233f4..da15556357 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755eb4..9bcefb6e6d 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb877..b1feff3e71 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2207,6 +2214,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9aff..0682c55b51 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c5c3..82d49428c2 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -197,6 +197,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
+/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9e63..6a7187bbec 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
2.21.0
0003-fixup-is_schema_sent-set-too-early-v3.patch (text/plain, 858B) download
From 473be553d5d798eac72700a5352d099295344cd6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH 03/17] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3e71..c0b97251e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1819,7 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
2.21.0
0004-Extend-the-output-plugin-API-with-stream-methods-v3.patch (text/plain, 34.8K) download
From c6f6ee18707a3dea988b98c6c5351d816a67be70 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/17] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 ++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 +++++
src/include/replication/reorderbuffer.h | 57 ++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4bded..9c77791dd5 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db968641e..fc4ad65eae 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615864..b88b58505a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e6d2..1e934d25e6 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -79,6 +79,11 @@ typedef struct LogicalDecodingContext
*/
void *output_writer_private;
+ /*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
/*
* State for writing output.
*/
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f26d..a30546250a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -99,6 +99,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
+/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
/*
* Output plugin callbacks
*/
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187bbec..5b4be2bf88 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -383,6 +429,17 @@ struct ReorderBuffer
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
+ /*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
/*
* Pointer that will be passed untouched to the callbacks.
*/
--
2.21.0
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structur-v3.patch (text/plain, 8.1K) download
From 8452286c8256ba59163f98309c7c59cefbe64845 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/17] Cleaning up of flags in ReorderBufferTXN structure
---
.../replication/logical/reorderbuffer.c | 36 +++++++++----------
src/include/replication/reorderbuffer.h | 33 ++++++++++-------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c0b97251e2..f74c1996d0 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1969,7 +1969,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2117,7 +2117,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2296,7 +2296,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2313,7 +2313,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2333,7 +2333,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2521,12 +2521,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2bf88..19c7bac8c0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -248,15 +264,6 @@ typedef struct ReorderBufferTXN
*/
uint64 nentries_mem;
- /*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
/*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
--
2.21.0
0006-Gracefully-handle-concurrent-aborts-of-uncommitte-v3.patch (text/plain, 12.5K) download
From 3656702c911785df27422ecd177e506723b0e33f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH 06/17] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 +-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++
.../replication/logical/reorderbuffer.c | 9 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65eae..da6a6f3233 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb34ef..2a60a7380a 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1421,6 +1432,16 @@ heap_fetch(Relation relation,
OffsetNumber offnum;
bool valid;
+ /*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
/*
* Fetch and pin the appropriate page of the relation.
*/
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1682,6 +1713,16 @@ heap_get_latest_tid(TableScanDesc sscan,
*/
Assert(ItemPointerIsValid(tid));
+ /*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
/*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d342..201acfbbf2 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f74c1996d0..76d2701233 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1876,6 +1876,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517596..9fa1e43347 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -153,6 +153,13 @@ static Snapshot SecondarySnapshot = NULL;
static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
+/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df48c..9a8f9ceba3 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
2.21.0
0007-Implement-streaming-mode-in-ReorderBuffer-v3.patch (text/plain, 44.3K) download
From 4dfb923fb4eea66cffb58de3175ae44fa56350b3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH 07/17] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
.../replication/logical/reorderbuffer.c | 1075 ++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e3646716f..cf10dd041d 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 76d2701233..f02c47238a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1234,6 +1319,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
pfree(state);
}
+/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1514,6 +1897,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_id = origin_id;
txn->origin_lsn = origin_lsn;
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
/*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1929,6 +2339,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2013,6 +2430,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2148,8 +2572,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2157,6 +2590,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2168,19 +2602,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2209,6 +2652,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2284,6 +2728,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2298,6 +2745,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2401,6 +2865,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
return largest;
}
+/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
@@ -2421,15 +2917,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2722,6 +3249,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac8c0..7d08e2fd39 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -221,6 +236,16 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
/*
* LSN pointing to the end of the commit record + 1.
*/
@@ -251,6 +276,13 @@ typedef struct ReorderBufferTXN
XLogRecPtr base_snapshot_lsn;
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
+ /*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
/*
* How many ReorderBufferChange's do we have in this txn.
*
--
2.21.0
0008-fixup-add-is_schema_sent-back-v3.patch (text/plain, 854B) download
From 6e512a0622a5a532aaa788b049ba028aa8fc0115 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:02:38 +0100
Subject: [PATCH 08/17] fixup: add is_schema_sent back
---
src/backend/replication/logical/reorderbuffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f02c47238a..0ab319182a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,6 +2229,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
2.21.0
0009-fixup-get-rid-of-is_schema_sent-entirely-v3.patch (text/plain, 3.5K) download
From 2a7baf771f161a165907b7f8038aed6d12e3cc36 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:46:15 +0100
Subject: [PATCH 09/17] fixup: get rid of is_schema_sent entirely
We'll do this in the pgoutput.c code directly, not in reorderbuffer.
---
.../replication/logical/reorderbuffer.c | 26 ++-----------------
src/include/replication/reorderbuffer.h | 5 ----
2 files changed, 2 insertions(+), 29 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab319182a..85db15ea3b 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,7 +2229,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
@@ -2729,9 +2728,6 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
-
- /* Invalidate current schema as well */
- txn->is_schema_sent = false;
}
/*
@@ -2747,22 +2743,12 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- /*
- * We read catalog changes from WAL, which are not yet sent, so
- * invalidate current schema in order output plugin can resend
- * schema again.
- */
- txn->is_schema_sent = false;
-
/*
* TOCHECK: Mark toplevel transaction as having catalog changes too
* if one of its children has.
*/
if (txn->toptxn != NULL)
- {
txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- txn->toptxn->is_schema_sent = false;
- }
}
/*
@@ -3345,9 +3331,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* TOCHECK: We have to rebuild historic snapshot to be sure it includes all
* information about subtransactions, which could arrive after streaming start.
*/
- if (!txn->is_schema_sent)
- snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
}
/*
@@ -3602,12 +3587,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
snapshot_now = change->data.snapshot;
}
- /*
- * TOCHECK: Snapshot changed, then invalidate current schema to reflect
- * possible catalog changes.
- */
- txn->is_schema_sent = false;
-
/* and continue with the new one */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
txn->xid);
@@ -3646,7 +3625,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2fd39..e2b8db0ff1 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -236,11 +236,6 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
- /*
- * Do we need to send schema for this transaction in output plugin?
- */
- bool is_schema_sent;
-
/*
* Toplevel transaction for this subxact (NULL for top-level).
*/
--
2.21.0
0010-Support-logical_decoding_work_mem-set-from-create-v3.patch (text/plain, 13.1K) download
From 129547a473edd42fa66f1490d36f8aaede529867 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH 10/17] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 ++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c90282f..8b1923c9de 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..91790b0c95 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff499..2a276482c1 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edcfc2..fbb447379f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fcd05..0ab6855ad8 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 63ba0ae234..c80acd3eb0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1729,6 +1729,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b877..cf6e03b9a7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..10ea113e4d 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714eaf0c..1db706af54 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.21.0
0011-Add-support-for-streaming-to-built-in-replication-v3.patch (text/plain, 89.5K) download
From 83135d7f8db59c80c2ab3b1a5c5542d9b0400fdb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH 11/17] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 +++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../t/012_stream_subxact_abort.pl | 82 ++
.../t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..e1fb9075e1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0c95..d9abf5e64c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a276482c1..15a6f5a8b3 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb447379f..b2b93d6234 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7410b2ff5e..a479ce9329 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855ad8..9970170e47 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index c57b578b48..0a013ed220 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b58505a..ad43ab365e 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..5a379fb6bc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c80acd3eb0..cf053e948b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
errmsg("ORIGIN message sent out of order")));
}
+/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
/*
* Handle RELATION message.
*
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -694,6 +1122,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -814,6 +1245,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -913,6 +1347,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1004,6 +1441,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1100,6 +1553,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
}
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
/*
* Apply main loop.
*/
@@ -1116,6 +1585,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1564,6 +2036,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1730,6 +2760,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b9a7..8490ea4717 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -512,6 +638,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
rel_sync_cache_publication_cb(arg, cacheid, hashvalue);
}
+/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
/*
* Initialize the relation schema sync cache for a decoding session.
*
@@ -622,6 +833,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
/*
* Relcache invalidation callback
*/
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ba08ad405f..8eb3160041 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1f23665432..2e0743ac8f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndPrepareWrite, WalSndWriteData,
WalSndUpdateProgress);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
/*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113e4d..8793676258 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2e873d048..c522703d8c 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..bf02cbc19d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706af54..3d19b5d88e 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000000..2f01133f69
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000000..d2ae38592b
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000000..0da39a1a8a
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000000..402df30f59
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000000..becbdd0578
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.21.0
0012-fixup-add-proper-schema-tracking-v3.patch (text/plain, 3.3K) download
From 7a45713c4161e4efdf82148f56af3328283b20e9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:56:04 +0100
Subject: [PATCH 12/17] fixup: add proper schema tracking
---
src/backend/replication/pgoutput/pgoutput.c | 45 ++++++++++++++++++++-
1 file changed, 43 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 8490ea4717..0148f4c01e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -82,6 +82,8 @@ typedef struct RelationSyncEntry
Oid relid; /* relation oid */
TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -96,6 +98,11 @@ static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -366,6 +373,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
bool schema_sent;
TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
/*
* Remember XID of the (sub)transaction for the change. We don't care if
@@ -378,6 +386,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (in_streaming)
xid = change->txn->xid;
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
/*
* Do we need to send the schema? We do track streamed transactions
* separately, because those may not be applied later (and the regular
@@ -391,7 +404,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
* occur when streaming already started, so we have to track new catalog
* changes somehow.
*/
- schema_sent = txn->is_schema_sent;
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
}
else
schema_sent = relentry->schema_sent;
@@ -432,7 +445,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->xid = change->txn->xid;
if (in_streaming)
- txn->is_schema_sent = true;
+ set_schema_sent_in_streamed_txn(relentry, topxid);
else
relentry->schema_sent = true;
}
@@ -759,6 +772,34 @@ init_rel_sync_cache(MemoryContext cachectx)
(Datum) 0);
}
+/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
/*
* Find or create entry in the relation schema cache.
*/
--
2.21.0
0013-Track-statistics-for-streaming-v3.patch (text/plain, 11.7K) download
From 0b9a94e41e81ef0ce58ada49da55cb10e2cc5a1f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH 13/17] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
.../replication/logical/reorderbuffer.c | 13 ++++++++
src/backend/replication/walsender.c | 32 ++++++++++++++++---
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 13 +++++---
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 ++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb58115af..180ea880a4 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..58976118db 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 85db15ea3b..65f876d0f0 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3709,6 +3713,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_END_TRY();
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
/*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2e0743ac8f..9f93f11d8a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3185,7 +3189,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3242,6 +3246,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3265,6 +3272,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3351,6 +3361,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3598,12 +3613,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b219..3b897a5d70 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e2b8db0ff1..e132c3c5ea 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -506,15 +506,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b32051ac..7efc332319 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a07825b9..5ab21a80d0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
2.21.0
0014-Enable-streaming-for-all-subscription-TAP-tests-v3.patch (text/plain, 14.8K) download
From 3b857c0da517c7aed243253e8411767a0e973d63 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH 14/17] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560b23..8cd1993393 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2fbc..94c71f8ae2 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f65fa..8dfeafc772 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9181..a6fae9c3f1 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17a78..202871a658 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10a19..70c86b22ac 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6d63..f9c8d1d348 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7332..0c9c6b3dd4 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bdba9..21f50c7012 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133f69..30561d8f96 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae38592b..9a6bac6822 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bdc35..ed56fbf96c 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cba4c..4df1ddef63 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1a8a..c3caff6149 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7c2f..c62eb521e7 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30f59..2be7542831 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0578..2da9607a7d 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9435..96ffc091b0 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
2.21.0
0015-BUGFIX-set-final_lsn-for-subxacts-before-cleanup-v3.patch (text/plain, 1013B) download
From 016b2a43955bb56f3a439d7a003fe9d0d6412125 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 15/17] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 65f876d0f0..5883d14a02 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
2.21.0
0016-Add-TAP-test-for-streaming-vs.-DDL-v3.patch (text/plain, 4.4K) download
From 9fd14e817ba569654dc32ef75aadde7300ad1b0f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 16/17] Add TAP test for streaming vs. DDL
---
.../subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000000..b8d78b1972
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.21.0
0017-Extend-handling-of-concurrent-aborts-for-streamin-v3.patch (text/plain, 3.7K) download
From c9d0c5da2641d0746657d465893bb65eb9d73b7e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH 17/17] Extend handling of concurrent aborts for streaming
transaction
---
.../replication/logical/reorderbuffer.c | 36 +++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 +++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5883d14a02..0987032208 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2349,9 +2349,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3267,6 +3267,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3396,6 +3397,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3457,6 +3465,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3695,6 +3707,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3713,7 +3728,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e132c3c5ea..6186465a85 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -236,6 +236,11 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
/*
* Toplevel transaction for this subxact (NULL for top-level).
*/
--
2.21.0
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-29 08:04 Dilip Kumar <dilipbalaut@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-12-29 08:04 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Dec 28, 2019 at 9:33 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Tue, Dec 10, 2019 at 10:23:19AM +0530, Dilip Kumar wrote:
> >On Tue, Dec 10, 2019 at 9:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >> >
> >> > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >> > >
> >> > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> >> > > > I have rebased the patch on the latest head and also fix the issue of
> >> > > > "concurrent abort handling of the (sub)transaction." and attached as
> >> > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> >> > > > the complete patch set. I have added the version number so that we
> >> > > > can track the changes.
> >> > >
> >> > > The patch has rotten a bit and does not apply anymore. Could you
> >> > > please send a rebased version? I have moved it to next CF, waiting on
> >> > > author.
> >> >
> >> > I have rebased the patch set on the latest head.
> >> >
> >> > Apart from this, there is one issue reported by my colleague Vignesh.
> >> > The issue is that if we use more than two relations in a transaction
> >> > then there is an error on standby (no relation map entry for remote
> >> > relation ID 16390). After analyzing I have found that for the
> >> > streaming transaction an "is_schema_sent" flag is kept in
> >> > ReorderBufferTXN. And, I think that is done so that we can send the
> >> > schema for each transaction stream so that if any subtransaction gets
> >> > aborted we don't lose the logical WAL for that schema. But, this
> >> > solution has induced a very basic issue that if a transaction operate
> >> > on more than 1 relation then after sending the schema for the first
> >> > relation it will mark the flag true and the schema for the subsequent
> >> > relations will never be sent.
> >> >
> >>
> >> How about keeping a list of top-level xids in each RelationSyncEntry?
> >> Basically, whenever we send the schema for any transaction, we note
> >> that in RelationSyncEntry and at abort time we can remove xid from the
> >> list. Now, whenever, we check whether to send schema for any
> >> operation in a transaction, we will check if our xid is present in
> >> that list for a particular RelationSyncEntry and take an action based
> >> on that (if xid is present, then we won't send the schema, otherwise,
> >> send it).
> >The idea make sense to me. I will try to write a patch for this and test.
> >
>
> Yeah, the "is_schema_sent" flag in ReorderBufferTXN does not work - it
> needs to be in the RelationSyncEntry. In fact, I already have code for
> that in my private repository - I thought the patches I sent here do
> include this, but apparently I forgot to include this bit :-(
>
> Attached is a rebased patch series, fixing this. It's essentially v2
> with a couple of patches (0003, 0008, 0009 and 0012) replacing the
> is_schema_sent with correct handling.
>
>
> 0003 - removes an is_schema_sent reference added prematurely (it's added
> by a later patch, causing compile failure)
>
> 0008 - adds the is_schema_sent back (essentially reverting 0003)
>
> 0009 - removes is_schema_sent entirely
>
> 0012 - adds the correct handling of schema flags in pgoutput
>
>
> I don't know what other changes you've made since v2, so this way it
> should be possible to just take 0003, 0008, 0009 and 0012 and slip them
> in with minimal hassle.
>
> FWIW thanks to everyone (and Amit and Dilip in particular) working on
> this patch series. There's been a lot of great reviews and improvements
> since I abandoned this thread for a while. I expect to be able to spend
> more time working on this in January.
>
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
I was looking into the schema tracking solution and I have one
question, Shouldn't we remove the topxid from the list if the
(sub)transaction is aborted? because once it is aborted we need to
resent the schema. I think we can remove the xid from the list in the
cleanup_rel_sync_cache function?
I have observed some more issues
1. Currently, In ReorderBufferCommit, it is always expected that
whenever we get REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM, we must
have already got REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT and in
SPEC_CONFIRM we send the tuple we got in SPECT_INSERT. But, now those
two messages can be in different streams. So we need to find a way to
handle this. Maybe once we get SPEC_INSERT then we can remember the
tuple and then if we get the SPECT_CONFIRM in the next stream we can
send that tuple?
2. During commit time in DecodeCommit we check whether we need to skip
the changes of the transaction or not by calling
SnapBuildXactNeedsSkip but since now we support streaming so it's
possible that before we decode the commit WAL, we might have already
sent the changes to the output plugin even though we could have
skipped those changes. So my question is instead of checking at the
commit time can't we check before adding to ReorderBuffer itself or we
can truncate the changes if SnapBuildXactNeedsSkip is true whenever
logical_decoding_workmem limit is reached. Am I missing something
here?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-30 09:40 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-12-30 09:40 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
Yesterday, Tomas has posted the latest version of the patch set which
contain the fix for schema send part. Meanwhile, I was working on few
review comments/bugfixes and refactoring. I have tried to merge those
changes with the latest patch set except the refactoring related to
"0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
has also made some changes in the same patch. I have created a
separate patch for the same so that we can review the changes and then
we can merge them to the main patch.
> On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have review the patch set and here are few comments/questions
> > >
> > > 1.
> > > +static void
> > > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > > + ReorderBufferTXN *txn,
> > > + Relation relation,
> > > + ReorderBufferChange *change)
> > > +{
> > > + OutputPluginPrepareWrite(ctx, true);
> > > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > > + OutputPluginWrite(ctx, true);
> > > +}
> > >
> > > Should we show the tuple in the streamed change like we do for the
> > > pg_decode_change?
> > >
> >
> > I think so. The patch shows the message in
> > pg_decode_stream_message(), so why to prohibit showing tuple here?
Yeah, we can do that. One option is that we can directly register
"pg_decode_change" function as stream_change_cb plugin and that will
show the tuple, another option is that we can write a similar function
as pg_decode_change and change the message which includes the text
"STREAM" so that the user can distinguish between tuple from committed
transaction and the in-progress transaction.
While analyzing this solution I have encountered one more issue, the
problem is that currently, during commit time in DecodeCommit we check
whether we need to skip the changes of the transaction or not by
calling SnapBuildXactNeedsSkip but since now we support streaming so
it's possible that before commit wal arrive we might have already sent
the changes to the output plugin even though we could have skipped
those changes. So my question is instead of checking at the commit
time can't we check before adding to ReorderBuffer itself or we can
truncate the changes if SnapBuildXactNeedsSkip is true whenever
logical_decoding_workmem limit is reached.
> > Few comments on this patch series:
> >
> > 0001-Immediately-WAL-log-assignments:
> > ------------------------------------------------------------
> >
> > The commit message still refers to the old design for this patch. I
> > think you need to modify the commit message as per the latest patch.
Done
> >
> > 0002-Issue-individual-invalidations-with-wal_level-log
> > ----------------------------------------------------------------------------
> > 1.
> > xact_desc_invalidations(StringInfo buf,
> > {
> > ..
> > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> >
> > You have removed logging for the above cache but forgot to remove its
> > reference from one of the places. Also, I think you need to add a
> > comment somewhere in inval.c to say why you are writing for WAL for
> > some types of invalidations and not for others?
Done
> >
> > 0003-Extend-the-output-plugin-API-with-stream-methods
> > --------------------------------------------------------------------------------
> > 1.
> > + are required, while <function>stream_message_cb</function> and
> > + <function>stream_message_cb</function> are optional.
> >
> > stream_message_cb is mentioned twice. It seems the second one is for truncate.
Done
> >
> > 2.
> > size of the transaction size and network bandwidth, the transfer time
> > + may significantly increase the apply lag.
> >
> > /size of the transaction size/size of the transaction
> >
> > no need to mention size twice.
Done
> >
> > 3.
> > + Similarly to spill-to-disk behavior, streaming is triggered when the total
> > + amount of changes decoded from the WAL (for all in-progress
> > transactions)
> > + exceeds limit defined by <varname>logical_work_mem</varname> setting.
> >
> > The guc name used is wrong. /Similarly to/Similar to/
Done
> >
> > 4.
> > stream_start_cb_wrapper()
> > {
> > ..
> > + /* state.report_location = apply_lsn; */
> > ..
> > + /* FIXME ctx->write_location = apply_lsn; */
> > ..
> > }
> >
> > See, if we can fix these and similar in the callback for the stop. I
> > think we don't have final_lsn till we commit/abort. Can we compute
> > before calling these API's?
Done
> >
> >
> > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > ----------------------------------------------------------------------------------
> > 1.
> > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > PG_CATCH();
> > {
> > /* TODO: Encapsulate cleanup
> > from the PG_TRY and PG_CATCH blocks */
> > +
> > if (iterstate)
> > ReorderBufferIterTXNFinish(rb, iterstate);
> >
> > Spurious line change.
> >
Done
> > 2. The commit message of this patch refers to Prepared transactions.
> > I think that needs to be changed.
> >
> > 0006-Implement-streaming-mode-in-ReorderBuffer
> > -------------------------------------------------------------------------
> > 1.
> > +
> > +/* iterator for streaming (only get data from memory) */
> > +static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
> > +
> > ReorderBuffer *rb,
> > +
> > ReorderBufferTXN
> > *txn);
> > +
> > +static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
> > + ReorderBuffer *rb,
> > +
> > ReorderBufferStreamIterTXNState * state);
> > +
> > +static void ReorderBufferStreamIterTXNFinish(
> > +
> > ReorderBuffer *rb,
> > +
> > ReorderBufferStreamIterTXNState * state);
> >
> > Do we really need to introduce new APIs for iterating over changes
> > from streamed transactions? Why can't we reuse the same API's as we
> > use for committed xacts?
Done
> >
> > 2.
> > +static void
> > +ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
> >
> > Please write some comments atop ReorderBufferStreamCommit.
Done
> >
> > 3.
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > ..
> > + if (txn->snapshot_now
> > == NULL)
> > + {
> > + dlist_iter subxact_i;
> > +
> > + /* make sure this transaction is streamed for the first time */
> > +
> > Assert(!rbtxn_is_streamed(txn));
> > +
> > + /* at the beginning we should have invalid command ID */
> > + Assert(txn->command_id ==
> > InvalidCommandId);
> > +
> > + dlist_foreach(subxact_i, &txn->subtxns)
> > + {
> > + ReorderBufferTXN *subtxn;
> > +
> > +
> > subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
> > +
> > + if (subtxn->base_snapshot != NULL &&
> > +
> > (txn->base_snapshot == NULL ||
> > + txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
> > + {
> > +
> > txn->base_snapshot = subtxn->base_snapshot;
> >
> > The logic here seems to be correct, but I am not sure why it is not
> > considered to purge the base snapshot before assigning the subtxn's
> > snapshot and similarly, we have not purged snapshot for subtxn once we
> > are done with it. I think we can use
> > ReorderBufferTransferSnapToParent to replace part of the logic here.
> > Do you see any reason for doing things differently here?
Done
> >
> > 4. In ReorderBufferStreamTXN, why do you need to use
> > ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
IMHO, here instead of directly copying the base snapshot we are
modifying it by passing command id and thats the reason we are copying
it.
> >
> > 5. I see a lot of code similarity in ReorderBufferStreamTXN and
> > existing ReorderBufferCommit. I understand that there are some subtle
> > differences due to which we need to write this new function but can't
> > we encapsulate the specific parts of code in functions and then call
> > from both places. I am talking about code in different cases for
> > change->action.
Done
> >
> > 6. + * Note: We never stream and serialize a transaction at the same time (e
> > /(e/(we
Done
I have also found one bug in
"v3-0012-fixup-add-proper-schema-tracking.patch" due to which some of
the streaming test cases were failing, I have created a separate patch
to fix the same.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v4-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 4fd022529dc2c4f5c4d96c0e69537ae5f40684e7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v4 01/19] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6a..708e523 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b0..4435c63 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3fea199..b1976ac 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611..a676151 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v4-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.7K) download
From c067b209589fa6288c6aef48955a21bd8ee2732d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v4 02/19] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 ++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 229 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..7ecdc0e 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e523..da15556 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v4-0003-fixup-is_schema_sent-set-too-early.patch (application/octet-stream, 858B) download
From 5c284042e2a21f7f7ed7bdc84a5eccec77e4c05f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH v4 03/19] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..c0b9725 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1819,7 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v4-0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 8a7ab4d4c83e2453ea926727c3c7985d4e0f7222 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v4 04/19] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..57edf54 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e..1e934d2 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v4-0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From 5ebe0327a929d10d4fa82f27e4828dc60c6e2701 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v4 05/19] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c0b9725..f74c199 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1969,7 +1969,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2117,7 +2117,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2296,7 +2296,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2313,7 +2313,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2333,7 +2333,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2521,12 +2521,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 11.9K) download
From 8b0ad280cae050bae5e6229cb3485967db919bb3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v4 06/19] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 ++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 109 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..a27eac0 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "improper heap_getnext call");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_fetch call");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_hot_search_buffer call");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_get_latest_tid call");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_hot_search call");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f74c199..3e87597 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v4-0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 4bc6427ef3439096a34d12dabb28ba60d09045d7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v4 07/19] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3e87597..232d9f4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1928,6 +2338,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2012,6 +2429,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2147,8 +2571,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2156,6 +2589,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2167,19 +2601,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2208,6 +2651,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2283,6 +2727,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2297,6 +2744,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2401,6 +2865,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2420,15 +2916,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2721,6 +3248,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v4-0008-fixup-add-is_schema_sent-back.patch (application/octet-stream, 854B) download
From dcb320769da1aa3c4bd7c80c3dcee22a0fb3ed33 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:02:38 +0100
Subject: [PATCH v4 08/19] fixup: add is_schema_sent back
---
src/backend/replication/logical/reorderbuffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 232d9f4..78b5c00 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,6 +2229,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v4-0009-fixup-get-rid-of-is_schema_sent-entirely.patch (application/octet-stream, 3.6K) download
From 1f6b7447ac3d80fc47a603d106b2d26bfc46c710 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:46:15 +0100
Subject: [PATCH v4 09/19] fixup: get rid of is_schema_sent entirely
We'll do this in the pgoutput.c code directly, not in reorderbuffer.
---
src/backend/replication/logical/reorderbuffer.c | 26 ++-----------------------
src/include/replication/reorderbuffer.h | 5 -----
2 files changed, 2 insertions(+), 29 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 78b5c00..dda651e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,7 +2229,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
@@ -2728,9 +2727,6 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
-
- /* Invalidate current schema as well */
- txn->is_schema_sent = false;
}
/*
@@ -2747,21 +2743,11 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
/*
- * We read catalog changes from WAL, which are not yet sent, so
- * invalidate current schema in order output plugin can resend
- * schema again.
- */
- txn->is_schema_sent = false;
-
- /*
* TOCHECK: Mark toplevel transaction as having catalog changes too
* if one of its children has.
*/
if (txn->toptxn != NULL)
- {
txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- txn->toptxn->is_schema_sent = false;
- }
}
/*
@@ -3344,9 +3330,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* TOCHECK: We have to rebuild historic snapshot to be sure it includes all
* information about subtransactions, which could arrive after streaming start.
*/
- if (!txn->is_schema_sent)
- snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
}
/*
@@ -3601,12 +3586,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
snapshot_now = change->data.snapshot;
}
- /*
- * TOCHECK: Snapshot changed, then invalidate current schema to reflect
- * possible catalog changes.
- */
- txn->is_schema_sent = false;
-
/* and continue with the new one */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
txn->xid);
@@ -3645,7 +3624,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..e2b8db0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -237,11 +237,6 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
- * Do we need to send schema for this transaction in output plugin?
- */
- bool is_schema_sent;
-
- /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
v4-0010-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 218d6f72deea3db0109db556086d73d0092cf9e7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v4 10/19] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c902..8b1923c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 63ba0ae..c80acd3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1729,6 +1729,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714ea..1db706a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v4-0011-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.6K) download
From 7e4f8b8f47289299f5b856f0e77925d4b8cbb166 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v4 11/19] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7410b2f..a479ce9 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index c57b578..0a013ed 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 57edf54..30a6ee4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c80acd3..cf053e9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -694,6 +1122,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -814,6 +1245,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -913,6 +1347,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1004,6 +1441,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1101,6 +1554,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1116,6 +1585,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1564,6 +2036,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1730,6 +2760,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ba08ad4..8eb3160 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1f23665..2e0743a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2e873d..c522703 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706a..3d19b5d 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v4-0012-fixup-add-proper-schema-tracking.patch (application/octet-stream, 3.3K) download
From 13f056ac3452588e6f8e3e832f222f3f6416857f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:56:04 +0100
Subject: [PATCH v4 12/19] fixup: add proper schema tracking
---
src/backend/replication/pgoutput/pgoutput.c | 45 +++++++++++++++++++++++++++--
1 file changed, 43 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 8490ea4..0148f4c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -82,6 +82,8 @@ typedef struct RelationSyncEntry
Oid relid; /* relation oid */
TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -96,6 +98,11 @@ static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -366,6 +373,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
bool schema_sent;
TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
/*
* Remember XID of the (sub)transaction for the change. We don't care if
@@ -378,6 +386,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (in_streaming)
xid = change->txn->xid;
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
/*
* Do we need to send the schema? We do track streamed transactions
* separately, because those may not be applied later (and the regular
@@ -391,7 +404,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
* occur when streaming already started, so we have to track new catalog
* changes somehow.
*/
- schema_sent = txn->is_schema_sent;
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
}
else
schema_sent = relentry->schema_sent;
@@ -432,7 +445,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->xid = change->txn->xid;
if (in_streaming)
- txn->is_schema_sent = true;
+ set_schema_sent_in_streamed_txn(relentry, topxid);
else
relentry->schema_sent = true;
}
@@ -760,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
--
1.8.3.1
v4-0013-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 88f482bd96c5ffbdc30e1dc76c79ddabd4ecd554 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH v4 13/19] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb5811..180ea88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dda651e..1949dc4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3709,6 +3713,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2e0743a..9f93f11 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3185,7 +3189,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3242,6 +3246,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3265,6 +3272,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3351,6 +3361,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3598,12 +3613,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b..3b897a5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e2b8db0..e132c3c 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -506,15 +506,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a0782..5ab21a8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v4-0014-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From a3d35c1aeb5b8dc4d2f810958966f5bc689bfd06 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v4 14/19] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v4-0015-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 441194d7823933ca81bf8cfb86d4780e91e166eb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v4 15/19] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1949dc4..a3e2c06 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v4-0016-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 403e8a2caee4f6ec7be98ac3c0ac8b60fb5af952 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v4 16/19] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 3.7K) download
From 4030d805e9358a374f522c2c11c0ac15e36e19a7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v4 17/19] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a3e2c06..789b425 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2348,9 +2348,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3266,6 +3266,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3395,6 +3396,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3456,6 +3464,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3694,6 +3706,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e132c3c..6186465 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -237,6 +237,11 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
v4-0018-Review-comment-fix-and-refactoring.patch (application/octet-stream, 36.1K) download
From 76cea7d3a261cde0492547db75e4c95ea787ad32 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Sun, 29 Dec 2019 15:41:26 +0530
Subject: [PATCH v4 18/19] Review comment fix and refactoring
---
src/backend/replication/logical/reorderbuffer.c | 995 ++++++------------------
1 file changed, 237 insertions(+), 758 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 789b425..8b3f112 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,28 +149,6 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
-/*
- * k-way in-order change iteration support structures
- *
- * This is a simplified version for streaming, which does not require
- * serialization to files and only reads changes that are currently in
- * memory.
- */
-typedef struct ReorderBufferStreamIterTXNEntry
-{
- XLogRecPtr lsn;
- ReorderBufferChange *change;
- ReorderBufferTXN *txn;
-} ReorderBufferStreamIterTXNEntry;
-
-typedef struct ReorderBufferStreamIterTXNState
-{
- binaryheap *heap;
- Size nr_txns;
- dlist_head old_change;
- ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
-} ReorderBufferStreamIterTXNState;
-
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -235,20 +213,6 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
-
-/* iterator for streaming (only get data from memory) */
-static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
- ReorderBuffer *rb,
- ReorderBufferTXN *txn);
-
-static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
- ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state);
-
-static void ReorderBufferStreamIterTXNFinish(
- ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state);
-
/*
* ---------------------------------------
* Disk serialization support functions
@@ -1324,210 +1288,6 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
- * Binary heap comparison function (streaming iterator).
- */
-static int
-ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
-{
- ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
- XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
- XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
-
- if (pos_a < pos_b)
- return 1;
- else if (pos_a == pos_b)
- return 0;
- return -1;
-}
-
-/*
- * Allocate & initialize an iterator which iterates in lsn order over a
- * transaction and all its subtransactions. This version is meant for
- * streaming of incomplete transactions.
- */
-static ReorderBufferStreamIterTXNState *
-ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
-{
- Size nr_txns = 0;
- ReorderBufferStreamIterTXNState *state;
- dlist_iter cur_txn_i;
- int32 off;
-
- /* Check ordering of changes in the toplevel transaction. */
- AssertChangeLsnOrder(rb, txn);
-
- /*
- * Calculate the size of our heap: one element for every transaction that
- * contains changes. (Besides the transactions already in the reorder
- * buffer, we count the one we were directly passed.)
- */
- if (txn->nentries > 0)
- nr_txns++;
-
- dlist_foreach(cur_txn_i, &txn->subtxns)
- {
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- /* Check ordering of changes in this subtransaction. */
- AssertChangeLsnOrder(rb, cur_txn);
-
- if (cur_txn->nentries > 0)
- nr_txns++;
- }
-
- /*
- * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
- * need to allocate/build a heap then.
- */
-
- /* allocate iteration state */
- state = (ReorderBufferStreamIterTXNState *)
- MemoryContextAllocZero(rb->context,
- sizeof(ReorderBufferStreamIterTXNState) +
- sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
-
- state->nr_txns = nr_txns;
- dlist_init(&state->old_change);
-
- /* allocate heap */
- state->heap = binaryheap_allocate(state->nr_txns,
- ReorderBufferStreamIterCompare,
- state);
-
- /*
- * Now insert items into the binary heap, in an unordered fashion. (We
- * will run a heap assembly step at the end; this is more efficient.)
- */
-
- off = 0;
-
- /* add toplevel transaction if it contains changes */
- if (txn->nentries > 0)
- {
- ReorderBufferChange *cur_change;
-
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &txn->changes);
-
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = txn;
-
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
- }
-
- /* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
- {
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- if (cur_txn->nentries > 0)
- {
- ReorderBufferChange *cur_change;
-
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &cur_txn->changes);
-
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = cur_txn;
-
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
- }
- }
-
- Assert(off == nr_txns);
-
- /* assemble a valid binary heap */
- binaryheap_build(state->heap);
-
- return state;
-}
-
-/*
- * Return the next change when iterating over a transaction and its
- * subtransactions.
- *
- * Returns NULL when no further changes exist.
- */
-static ReorderBufferChange *
-ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
-{
- ReorderBufferChange *change;
- ReorderBufferStreamIterTXNEntry *entry;
- int32 off;
-
- /* nothing there anymore */
- if (state->heap->bh_size == 0)
- return NULL;
-
- off = DatumGetInt32(binaryheap_first(state->heap));
- entry = &state->entries[off];
-
- /* free memory we might have "leaked" in the previous *Next call */
- if (!dlist_is_empty(&state->old_change))
- {
- change = dlist_container(ReorderBufferChange, node,
- dlist_pop_head_node(&state->old_change));
- ReorderBufferReturnChange(rb, change);
- Assert(dlist_is_empty(&state->old_change));
- }
-
- change = entry->change;
-
- /*
- * update heap with information about which transaction has the next
- * relevant change in LSN order
- */
-
- /* there are in-memory changes */
- if (dlist_has_next(&entry->txn->changes, &entry->change->node))
- {
- dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
- ReorderBufferChange *next_change =
- dlist_container(ReorderBufferChange, node, next);
-
- /* txn stays the same */
- state->entries[off].lsn = next_change->lsn;
- state->entries[off].change = next_change;
-
- binaryheap_replace_first(state->heap, Int32GetDatum(off));
- return change;
- }
-
- /* ok, no changes there anymore, remove */
- binaryheap_remove_first(state->heap);
-
- return change;
-}
-
-/*
- * Deallocate the iterator
- */
-static void
-ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state)
-{
- /* free memory we might have "leaked" in the last *Next call */
- if (!dlist_is_empty(&state->old_change))
- {
- ReorderBufferChange *change;
-
- change = dlist_container(ReorderBufferChange, node,
- dlist_pop_head_node(&state->old_change));
- ReorderBufferReturnChange(rb, change);
- Assert(dlist_is_empty(&state->old_change));
- }
-
- binaryheap_free(state->heap);
- pfree(state);
-}
-
-/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1854,6 +1614,11 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ */
static void
ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
@@ -1868,86 +1633,38 @@ ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
*
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
+ Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
-
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
- * If the transaction was (partially) streamed, we need to commit it in a
- * 'streamed' way. That is, we first stream the remaining part of the
- * transaction, and then invoke stream_commit message.
- *
- * XXX Called after everything (origin ID and LSN, ...) is stored in the
- * transaction, so we don't pass that directly.
- *
- * XXX Somewhat hackish redirection, perhaps needs to be refactored?
- */
- if (rbtxn_is_streamed(txn))
- {
- ReorderBufferStreamCommit(rb, txn);
- return;
- }
-
- /*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
- */
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
- ReorderBufferBuildTupleCidHash(rb, txn);
-
- /* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
-
- /*
- * Decoding needs access to syscaches et al., which in turn use
- * heavyweight locks and such. Thus we need to have enough state around to
- * keep track of those. The easiest way is to simply use a transaction
- * internally. That also allows us to easily enforce that nothing writes
- * to the database by checking for xid assignments.
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
*
* When we're called via the SQL SRF there's already a transaction
* started, so start an explicit subtransaction there.
@@ -1961,11 +1678,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
iterstate = ReorderBufferIterTXNInit(rb, txn);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1978,11 +1699,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* subtransactions. The changes may have the same LSN due to
* MULTI_INSERT xlog records.
*/
- if (prev_lsn != InvalidXLogRecPtr)
- Assert(prev_lsn <= change->lsn);
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
prev_lsn = change->lsn;
+ if (streaming)
+ {
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
+ /* Increment the stream count. */
+ streamed++;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1992,8 +1725,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -2059,7 +1790,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -2080,8 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -2099,7 +1836,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -2157,7 +1894,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -2166,10 +1911,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -2200,9 +1951,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -2222,7 +1973,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -2260,16 +2012,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
- /* cleanup */
- TeardownHistoricSnapshot(false);
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
/*
* Aborting the current (sub-)transaction as a whole has the right
@@ -2285,14 +2067,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -2311,18 +2101,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -3264,10 +3153,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
volatile Snapshot snapshot_now;
volatile CommandId command_id;
- bool using_subtxn;
- Size streamed = 0;
- MemoryContext ccxt = CurrentMemoryContext;
- ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
* If this is a subxact, we need to stream the top-level transaction
@@ -3305,16 +3190,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferTXN *subtxn;
subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
-
- if (subtxn->base_snapshot != NULL &&
- (txn->base_snapshot == NULL ||
- txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
- {
- txn->base_snapshot = subtxn->base_snapshot;
- txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
- subtxn->base_snapshot = NULL;
- subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
- }
+ ReorderBufferTransferSnapToParent(txn, subtxn);
}
command_id = FirstCommandId;
@@ -3344,407 +3220,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * build data to be able to lookup the CommandIds of catalog tuples
- */
- ReorderBufferBuildTupleCidHash(rb, txn);
-
- /* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
-
- /*
- * Decoding needs access to syscaches et al., which in turn use
- * heavyweight locks and such. Thus we need to have enough state around to
- * keep track of those. The easiest way is to simply use a transaction
- * internally. That also allows us to easily enforce that nothing writes
- * to the database by checking for xid assignments.
- *
- * When we're called via the SQL SRF there's already a transaction
- * started, so start an explicit subtransaction there.
+ * Access the main routine to decode the changes and send to output plugin.
*/
- using_subtxn = IsTransactionOrTransactionBlock();
-
- PG_TRY();
- {
- XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- ReorderBufferChange *change;
- ReorderBufferChange *specinsert = NULL;
-
- if (using_subtxn)
- BeginInternalSubTransaction("stream");
- else
- StartTransactionCommand();
-
- /* start streaming this chunk of transaction */
- rb->stream_start(rb, txn);
-
- iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
- while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
- {
- Relation relation = NULL;
- Oid reloid;
-
- /*
- * Enforce correct ordering of changes, merged from multiple
- * subtransactions. The changes may have the same LSN due to
- * MULTI_INSERT xlog records.
- */
- if (prev_lsn != InvalidXLogRecPtr)
- Assert(prev_lsn <= change->lsn);
-
- prev_lsn = change->lsn;
-
- /* we're going to stream this change */
- streamed++;
-
- /*
- * Set the CheckXidAlive to the current (sub)xid for which this
- * change belongs to so that we can detect the abort while we are
- * decoding.
- */
- CheckXidAlive = change->txn->xid;
-
- switch (change->action)
- {
- case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
-
- /*
- * Confirmation for speculative insertion arrived. Simply
- * use as a normal record. It'll be cleaned up at the end
- * of INSERT processing.
- */
- Assert(specinsert->data.tp.oldtuple == NULL);
- change = specinsert;
- change->action = REORDER_BUFFER_CHANGE_INSERT;
-
- /* intentionally fall through */
- case REORDER_BUFFER_CHANGE_INSERT:
- case REORDER_BUFFER_CHANGE_UPDATE:
- case REORDER_BUFFER_CHANGE_DELETE:
- Assert(snapshot_now);
-
- reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
- change->data.tp.relnode.relNode);
-
- /*
- * Catalog tuple without data, emitted while catalog was
- * in the process of being rewritten.
- */
- if (reloid == InvalidOid &&
- change->data.tp.newtuple == NULL &&
- change->data.tp.oldtuple == NULL)
- goto change_done;
- else if (reloid == InvalidOid)
- elog(ERROR, "could not map filenode \"%s\" to relation OID",
- relpathperm(change->data.tp.relnode,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
-
- if (relation == NULL)
- elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
- reloid,
- relpathperm(change->data.tp.relnode,
- MAIN_FORKNUM));
-
- if (!RelationIsLogicallyLogged(relation))
- goto change_done;
-
- /*
- * For now ignore sequence changes entirely. Most of the
- * time they don't log changes using records we
- * understand, so it doesn't make sense to handle the few
- * cases we do.
- */
- if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
- goto change_done;
-
- /* user-triggered change */
- if (!IsToastRelation(relation))
- {
- ReorderBufferToastReplace(rb, txn, relation, change);
- rb->stream_change(rb, txn, relation, change);
-
- /* Remember that we have sent some data for this txn.*/
- if (!change->txn->any_data_sent)
- change->txn->any_data_sent = true;
-
- /*
- * Only clear reassembled toast chunks if we're sure
- * they're not required anymore. The creator of the
- * tuple tells us.
- */
- if (change->data.tp.clear_toast_afterwards)
- ReorderBufferToastReset(rb, txn);
- }
- /* we're not interested in toast deletions */
- else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
- {
- /*
- * Need to reassemble the full toasted Datum in
- * memory, to ensure the chunks don't get reused till
- * we're done remove it from the list of this
- * transaction's changes. Otherwise it will get
- * freed/reused while restoring spooled data from
- * disk.
- */
- dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
- }
-
- change_done:
-
- /*
- * Either speculative insertion was confirmed, or it was
- * unsuccessful and the record isn't needed anymore.
- */
- if (specinsert != NULL)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- if (relation != NULL)
- {
- RelationClose(relation);
- relation = NULL;
- }
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
-
- /*
- * Speculative insertions are dealt with by delaying the
- * processing of the insert until the confirmation record
- * arrives. For that we simply unlink the record from the
- * chain, so it does not get freed/reused while restoring
- * spooled data from disk.
- *
- * This is safe in the face of concurrent catalog changes
- * because the relevant relation can't be changed between
- * speculative insertion and confirmation due to
- * CheckTableNotInUse() and locking.
- */
-
- /* clear out a pending (and thus failed) speculation */
- if (specinsert != NULL)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- /* and memorize the pending insertion */
- dlist_delete(&change->node);
- specinsert = change;
- break;
-
- case REORDER_BUFFER_CHANGE_TRUNCATE:
- {
- int i;
- int nrelids = change->data.truncate.nrelids;
- int nrelations = 0;
- Relation *relations;
-
- relations = palloc0(nrelids * sizeof(Relation));
- for (i = 0; i < nrelids; i++)
- {
- Oid relid = change->data.truncate.relids[i];
- Relation relation;
-
- relation = RelationIdGetRelation(relid);
-
- if (relation == NULL)
- elog(ERROR, "could not open relation with OID %u", relid);
-
- if (!RelationIsLogicallyLogged(relation))
- continue;
-
- relations[nrelations++] = relation;
- }
-
- rb->stream_truncate(rb, txn, nrelations, relations, change);
-
- for (i = 0; i < nrelations; i++)
- RelationClose(relations[i]);
-
- break;
- }
-
- case REORDER_BUFFER_CHANGE_MESSAGE:
-
- rb->stream_message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
- /* get rid of the old */
- TeardownHistoricSnapshot(false);
-
- if (snapshot_now->copied)
- {
- ReorderBufferFreeSnap(rb, snapshot_now);
- snapshot_now =
- ReorderBufferCopySnap(rb, change->data.snapshot,
- txn, command_id);
- }
-
- /*
- * Restored from disk, need to be careful not to double
- * free. We could introduce refcounting for that, but for
- * now this seems infrequent enough not to care.
- */
- else if (change->data.snapshot->copied)
- {
- snapshot_now =
- ReorderBufferCopySnap(rb, change->data.snapshot,
- txn, command_id);
- }
- else
- {
- snapshot_now = change->data.snapshot;
- }
-
- /* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
- txn->xid);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
- Assert(change->data.command_id != InvalidCommandId);
-
- if (command_id < change->data.command_id)
- {
- command_id = change->data.command_id;
-
- if (!snapshot_now->copied)
- {
- /* we don't use the global one anymore */
- snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
- }
-
- snapshot_now->curcid = command_id;
-
- TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
- txn->xid);
- }
-
- break;
-
- case REORDER_BUFFER_CHANGE_INVALIDATION:
-
- /*
- * Execute the invalidation message locally.
- *
- * XXX Do we need to care about relcacheInitFileInval and
- * the other fields added to ReorderBufferChange, or just
- * about the message itself?
- */
- LocalExecuteInvalidationMessage(&change->data.inval.msg);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
- elog(ERROR, "tuplecid value in changequeue");
- break;
- }
- }
-
- /*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
- */
- if (specinsert)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- /* clean up the iterator */
- ReorderBufferStreamIterTXNFinish(rb, iterstate);
- iterstate = NULL;
-
- /* call stream_stop callback */
- rb->stream_stop(rb, txn);
-
- /* this is just a sanity check against bad output plugin behaviour */
- if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
- elog(ERROR, "output plugin used XID %u",
- GetCurrentTransactionId());
-
- /* remember the command ID and snapshot for the streaming run */
- txn->command_id = command_id;
- txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
-
- /* cleanup */
- TeardownHistoricSnapshot(false);
-
- /*
- * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
- * any memory. We could also keep the hash table and update it with
- * new ctid values, but this seems simpler and good enough for now.
- */
- ReorderBufferDestroyTupleCidHash(rb, txn);
-
- /*
- * Aborting the current (sub-)transaction as a whole has the right
- * semantics. We want all locks acquired in here to be released, not
- * reassigned to the parent and we do not want any database access
- * have persistent effects.
- */
- AbortCurrentTransaction();
-
- /* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
-
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
- }
- PG_CATCH();
- {
- MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
- ErrorData *errdata = CopyErrorData();
-
- /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
- if (iterstate)
- ReorderBufferStreamIterTXNFinish(rb, iterstate);
-
- TeardownHistoricSnapshot(true);
-
- /*
- * Force cache invalidation to happen outside of a valid transaction
- * to prevent catalog access as we just caught an error.
- */
- AbortCurrentTransaction();
-
- /* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
-
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
-
- /* re-throw only if it's not an abort */
- if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
- {
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
- }
- else
- {
- /* remember the command ID and snapshot for the streaming run */
- txn->command_id = command_id;
- txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
- rb->stream_stop(rb, txn);
-
- FlushErrorState();
- }
- }
- PG_END_TRY();
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
/*
* Update the stream statistics.
--
1.8.3.1
v4-0019-Bugfix-in-schema-tracking.patch (application/octet-stream, 837B) download
From 036f61804046977873257075143af41b7abe2875 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 30 Dec 2019 13:14:25 +0530
Subject: [PATCH v4 19/19] Bugfix in schema-tracking
---
src/backend/replication/pgoutput/pgoutput.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 0148f4c..ffd9f94 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -937,7 +937,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-30 10:13 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-30 10:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Dec 29, 2019 at 1:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have observed some more issues
>
> 1. Currently, In ReorderBufferCommit, it is always expected that
> whenever we get REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM, we must
> have already got REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT and in
> SPEC_CONFIRM we send the tuple we got in SPECT_INSERT. But, now those
> two messages can be in different streams. So we need to find a way to
> handle this. Maybe once we get SPEC_INSERT then we can remember the
> tuple and then if we get the SPECT_CONFIRM in the next stream we can
> send that tuple?
>
Your suggestion makes sense to me. So, we can try it.
> 2. During commit time in DecodeCommit we check whether we need to skip
> the changes of the transaction or not by calling
> SnapBuildXactNeedsSkip but since now we support streaming so it's
> possible that before we decode the commit WAL, we might have already
> sent the changes to the output plugin even though we could have
> skipped those changes. So my question is instead of checking at the
> commit time can't we check before adding to ReorderBuffer itself
>
I think if we can do that then the same will be true for current code
irrespective of this patch. I think it is possible that we can't take
that decision while decoding because we haven't assembled a consistent
snapshot yet. I think we might be able to do that while we try to
stream the changes. I think we need to take care of all the
conditions during streaming (when the logical_decoding_workmem limit
is reached) as we do in DecodeCommit. This needs a bit more study.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2019-12-30 10:19 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-30 10:19 UTC (permalink / raw)
To: Masahiko Sawada <masahiko.sawada@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 26, 2019 at 12:36 PM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Tue, 24 Dec 2019 at 17:21, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > >
> > > Thank you for explanation. The plan makes sense. But I think in the
> > > current design it's a problem that logical replication worker doesn't
> > > receive changes (and doesn't check interrupts) during applying
> > > committed changes even if we don't have a worker dedicated for
> > > applying. I think the worker should continue to receive changes and
> > > save them to temporary files even during applying changes.
> > >
> >
> > Won't it beat the purpose of this feature which is to reduce the apply
> > lag? Basically, it can so happen that while applying commit, it
> > constantly gets changes of other transactions which will delay the
> > apply of the current transaction.
>
> You're right. But it seems to me that it optimizes the apply lags of
> only a transaction that made many changes. On the other hand if a
> transaction made many changes applying of subsequent changes are
> delayed.
>
Hmm, how would it be worse than the current situation where once
commit is encountered on the publisher, we won't start with other
transactions until the replay of the same is finished on subscriber?
>
> > I think the best way as
> > discussed is to launch new workers for streamed transactions, but we
> > can do that as an additional feature. Anyway, as proposed, users can
> > choose the streaming mode for subscriptions, so there is an option to
> > turn this selectively.
>
> Yes. But user who wants to use this feature would want to replicate
> many changes but I guess the side effect is quite big. I think that at
> least we need to make the logical replication tolerate such situation.
>
What exactly you mean by "at least we need to make the logical
replication tolerate such situation."? Do you have something specific
in mind?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-02 05:50 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-02 05:50 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 30, 2019 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Dec 29, 2019 at 1:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have observed some more issues
> >
> > 1. Currently, In ReorderBufferCommit, it is always expected that
> > whenever we get REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM, we must
> > have already got REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT and in
> > SPEC_CONFIRM we send the tuple we got in SPECT_INSERT. But, now those
> > two messages can be in different streams. So we need to find a way to
> > handle this. Maybe once we get SPEC_INSERT then we can remember the
> > tuple and then if we get the SPECT_CONFIRM in the next stream we can
> > send that tuple?
> >
>
> Your suggestion makes sense to me. So, we can try it.
Sure.
>
> > 2. During commit time in DecodeCommit we check whether we need to skip
> > the changes of the transaction or not by calling
> > SnapBuildXactNeedsSkip but since now we support streaming so it's
> > possible that before we decode the commit WAL, we might have already
> > sent the changes to the output plugin even though we could have
> > skipped those changes. So my question is instead of checking at the
> > commit time can't we check before adding to ReorderBuffer itself
> >
>
> I think if we can do that then the same will be true for current code
> irrespective of this patch. I think it is possible that we can't take
> that decision while decoding because we haven't assembled a consistent
> snapshot yet. I think we might be able to do that while we try to
> stream the changes. I think we need to take care of all the
> conditions during streaming (when the logical_decoding_workmem limit
> is reached) as we do in DecodeCommit. This needs a bit more study.
I agree.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-03 09:24 Amit Kapila <amit.kapila16@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-03 09:24 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Dec 24, 2019 at 10:58 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Dec 12, 2019 at 3:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > I think the way invalidations work for logical replication is that
> > normally, we always start a new transaction before decoding each
> > commit which allows us to accept the invalidations (via
> > AtStart_Cache). However, if there are catalog changes within the
> > transaction being decoded, we need to reflect those before trying to
> > decode the WAL of operation which happened after that catalog change.
> > As we are not logging the WAL for each invalidation, we need to
> > execute all the invalidation messages for this transaction at each
> > catalog change. We are able to do that now as we decode the entire WAL
> > for a transaction only once we get the commit's WAL which contains all
> > the invalidation messages. So, we queue them up and execute them for
> > each catalog change which we identify by WAL record
> > XLOG_HEAP2_NEW_CID.
>
> Thanks for the explanation. That makes sense. But, it's still true,
> AFAICS, that instead of doing this stuff with logging invalidations
> you could just InvalidateSystemCaches() in the cases where you are
> currently applying all of the transaction's invalidations. That
> approach might be worse than changing the way invalidations are
> logged, but the two approaches deserve to be compared. One approach
> has more CPU overhead and the other has more WAL overhead, so it's a
> little hard to compare them, but it seems worth mulling over.
>
I have given some thought over it and it seems to me that this will
increase not only CPU usage but also Network usage. The increase in
CPU usage will be for all WALSenders that decodes a transaction that
has performed DDL. The increase in network usage comes from the fact
that we need to send the schema of relations again which doesn't
require to be invalidated. It is because invalidation blew our local
map that remembers which relation schemas are sent.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-04 04:30 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-04 04:30 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Dec 29, 2019 at 1:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> On Sat, Dec 28, 2019 at 9:33 PM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> >
> > Yeah, the "is_schema_sent" flag in ReorderBufferTXN does not work - it
> > needs to be in the RelationSyncEntry. In fact, I already have code for
> > that in my private repository - I thought the patches I sent here do
> > include this, but apparently I forgot to include this bit :-(
> >
> > Attached is a rebased patch series, fixing this. It's essentially v2
> > with a couple of patches (0003, 0008, 0009 and 0012) replacing the
> > is_schema_sent with correct handling.
> >
> >
> > 0003 - removes an is_schema_sent reference added prematurely (it's added
> > by a later patch, causing compile failure)
> >
> > 0008 - adds the is_schema_sent back (essentially reverting 0003)
> >
> > 0009 - removes is_schema_sent entirely
> >
> > 0012 - adds the correct handling of schema flags in pgoutput
> >
Thanks for splitting the changes. They are quite clear.
> >
> > I don't know what other changes you've made since v2, so this way it
> > should be possible to just take 0003, 0008, 0009 and 0012 and slip them
> > in with minimal hassle.
> >
> > FWIW thanks to everyone (and Amit and Dilip in particular) working on
> > this patch series. There's been a lot of great reviews and improvements
> > since I abandoned this thread for a while. I expect to be able to spend
> > more time working on this in January.
> >
> +static void
> +set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
> +{
> + MemoryContext oldctx;
> +
> + oldctx = MemoryContextSwitchTo(CacheMemoryContext);
> +
> + entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
> +
> + MemoryContextSwitchTo(oldctx);
> +}
> I was looking into the schema tracking solution and I have one
> question, Shouldn't we remove the topxid from the list if the
> (sub)transaction is aborted? because once it is aborted we need to
> resent the schema.
>
I think you are right because, at abort, the subscriber would remove
the changes (for a subtransaction) including the schema changes sent
and then it won't be able to understand the subsequent changes sent by
the publisher. Won't we need to remove xid from the list at commit
time as well, otherwise, the list will keep on growing. One more
thing, we need to search the list of all the relations in the local
map to find xid being aborted/committed, right? If so, won't it be
costly doing at each transaction abort/commit?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-04 06:24 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-04 06:24 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 4, 2020 at 10:00 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Dec 29, 2019 at 1:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > On Sat, Dec 28, 2019 at 9:33 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > +static void
> > +set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
> > +{
> > + MemoryContext oldctx;
> > +
> > + oldctx = MemoryContextSwitchTo(CacheMemoryContext);
> > +
> > + entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
> > +
> > + MemoryContextSwitchTo(oldctx);
> > +}
> > I was looking into the schema tracking solution and I have one
> > question, Shouldn't we remove the topxid from the list if the
> > (sub)transaction is aborted? because once it is aborted we need to
> > resent the schema.
> >
>
> I think you are right because, at abort, the subscriber would remove
> the changes (for a subtransaction) including the schema changes sent
> and then it won't be able to understand the subsequent changes sent by
> the publisher. Won't we need to remove xid from the list at commit
> time as well, otherwise, the list will keep on growing.
Yes, we need to remove the xid from the list at the time of commit as well.
One more
> thing, we need to search the list of all the relations in the local
> map to find xid being aborted/committed, right? If so, won't it be
> costly doing at each transaction abort/commit?
Yeah, if multiple concurrent transactions operate on the common
relations then the list can grow longer. I am not sure how many
concurrent large transactions are possible maybe it won't be huge that
searching will be very costly. Otherwise, we can maintain the sorted
array of the xids and do a binary search or we can maintain hash?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-04 10:37 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 3 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-04 10:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 30, 2019 at 3:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> Yesterday, Tomas has posted the latest version of the patch set which
> contain the fix for schema send part. Meanwhile, I was working on few
> review comments/bugfixes and refactoring. I have tried to merge those
> changes with the latest patch set except the refactoring related to
> "0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
> has also made some changes in the same patch.
>
I don't see any changes by Tomas in that particular patch, am I
missing something?
> I have created a
> separate patch for the same so that we can review the changes and then
> we can merge them to the main patch.
>
It is better to merge it with the main patch for
"Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
difficult to review.
> > > 0002-Issue-individual-invalidations-with-wal_level-log
> > > ----------------------------------------------------------------------------
> > > 1.
> > > xact_desc_invalidations(StringInfo buf,
> > > {
> > > ..
> > > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> > >
> > > You have removed logging for the above cache but forgot to remove its
> > > reference from one of the places. Also, I think you need to add a
> > > comment somewhere in inval.c to say why you are writing for WAL for
> > > some types of invalidations and not for others?
> Done
>
I don't see any new comments as asked by me. I think we should also
consider WAL logging at each command end instead of doing piecemeal as
discussed in another email [1], which will have lesser code changes
and maybe better in performance. You might want to evaluate the
performance of both approaches.
> > >
> > > 0003-Extend-the-output-plugin-API-with-stream-methods
> > > --------------------------------------------------------------------------------
> > >
> > > 4.
> > > stream_start_cb_wrapper()
> > > {
> > > ..
> > > + /* state.report_location = apply_lsn; */
> > > ..
> > > + /* FIXME ctx->write_location = apply_lsn; */
> > > ..
> > > }
> > >
> > > See, if we can fix these and similar in the callback for the stop. I
> > > think we don't have final_lsn till we commit/abort. Can we compute
> > > before calling these API's?
> Done
>
You have just used final_lsn, but I don't see where you have ensured
that it is set before the API stream_stop_cb_wrapper. I think we need
something similar to what Vignesh has done in one of his bug-fix patch
[2]. See my comment below in this regard.
> > >
> > >
> > > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > > ----------------------------------------------------------------------------------
> > > 1.
> > > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > > PG_CATCH();
> > > {
> > > /* TODO: Encapsulate cleanup
> > > from the PG_TRY and PG_CATCH blocks */
> > > +
> > > if (iterstate)
> > > ReorderBufferIterTXNFinish(rb, iterstate);
> > >
> > > Spurious line change.
> > >
> Done
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "improper heap_getnext call");
Earlier, I thought we don't need to check if it is a regular table in
this check, but it is required because output plugins can try to do
that and if they do so during decoding (with historic snapshots), the
same should be not allowed.
How about changing the error message to "unexpected heap_getnext call
during logical decoding" or something like that?
> > > 2. The commit message of this patch refers to Prepared transactions.
> > > I think that needs to be changed.
> > >
> > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > -------------------------------------------------------------------------
Few comments on v4-0018-Review-comment-fix-and-refactoring:
1.
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
2.
+ if (streaming)
+ {
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
+ /* Increment the stream count. */
+ streamed++;
+ }
Is the variable 'streamed' used anywhere?
3.
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
Won't this be required only when we are streaming changes?
As per my understanding apart from the above comments, the known
pending work for this patchset is as follows:
a. The two open items agreed to you in the email [3].
b. Complete the handling of schema_sent as discussed above [4].
c. Few comments by Vignesh and the response on the same by me [5][6].
d. WAL overhead and performance testing for additional WAL logging by
this patchset.
e. Some way to see the tuple for streamed transactions by decoding API
as speculated by you [7].
Have I missed anything?
[1] - https://www.postgresql.org/message-id/CAA4eK1LOa%2B2KqNX%3Dm%3D1qMBDW%2Bo50AuwjAOX6ZqL-rWGiH1F9MQ%40...
[2] - https://www.postgresql.org/message-id/CALDaNm3MDxFnsZsnSqVhPBLS3%3DqzNH6%2BYzB%3DxYuX2vbtsUeFgw%40ma...
[3] - https://www.postgresql.org/message-id/CAFiTN-uT5YZE0egGhKdTteTjcGrPi8hb%3DFMPpr9_hEB7hozQ-Q%40mail.g...
[4] - https://www.postgresql.org/message-id/CAA4eK1KjD9x0mS4JxzCbu3gu-r6K7XJRV%2BZcGb3BH6U3x2uxew%40mail.g...
[5] - https://www.postgresql.org/message-id/CALDaNm0DNUojjt7CV-fa59_kFbQQ3rcMBtauvo44ttea7r9KaA%40mail.gma...
[6] - https://www.postgresql.org/message-id/CAA4eK1%2BZvupW00c--dqEg8f3dHZDOGmA9xOQLyQHjRSoDi6AkQ%40mail.g...
[7] - https://www.postgresql.org/message-id/CAFiTN-t8PmKA1X4jEqKmkvs0ggWpy0APWpPuaJwpx2YpfAf97w%40mail.gma...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-06 03:50 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 03:50 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 30, 2019 at 3:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > Yesterday, Tomas has posted the latest version of the patch set which
> > contain the fix for schema send part. Meanwhile, I was working on few
> > review comments/bugfixes and refactoring. I have tried to merge those
> > changes with the latest patch set except the refactoring related to
> > "0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
> > has also made some changes in the same patch.
> >
>
> I don't see any changes by Tomas in that particular patch, am I
> missing something?
He has created some sub-patch from the main patch for handling
schema-sent issue. So if I make change in that patch all other
patches will conflict.
>
> > I have created a
> > separate patch for the same so that we can review the changes and then
> > we can merge them to the main patch.
> >
>
> It is better to merge it with the main patch for
> "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> difficult to review.
Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
(0007). Basically, if we merge all of them then we don't need to deal
with the conflict. I think Tomas has kept them separate so that we
can review the solution for the schema sent. And, I kept 0018 as a
separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
In the next patch set, I will merge all of them to 0007.
>
> > > > 0002-Issue-individual-invalidations-with-wal_level-log
> > > > ----------------------------------------------------------------------------
> > > > 1.
> > > > xact_desc_invalidations(StringInfo buf,
> > > > {
> > > > ..
> > > > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > > > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> > > >
> > > > You have removed logging for the above cache but forgot to remove its
> > > > reference from one of the places. Also, I think you need to add a
> > > > comment somewhere in inval.c to say why you are writing for WAL for
> > > > some types of invalidations and not for others?
> > Done
> >
>
> I don't see any new comments as asked by me.
Oh, I just fixed one part of the comment and overlooked the rest. Will fix.
I think we should also
> consider WAL logging at each command end instead of doing piecemeal as
> discussed in another email [1], which will have lesser code changes
> and maybe better in performance. You might want to evaluate the
> performance of both approaches.
Ok
>
> > > >
> > > > 0003-Extend-the-output-plugin-API-with-stream-methods
> > > > --------------------------------------------------------------------------------
> > > >
> > > > 4.
> > > > stream_start_cb_wrapper()
> > > > {
> > > > ..
> > > > + /* state.report_location = apply_lsn; */
> > > > ..
> > > > + /* FIXME ctx->write_location = apply_lsn; */
> > > > ..
> > > > }
> > > >
> > > > See, if we can fix these and similar in the callback for the stop. I
> > > > think we don't have final_lsn till we commit/abort. Can we compute
> > > > before calling these API's?
> > Done
> >
>
> You have just used final_lsn, but I don't see where you have ensured
> that it is set before the API stream_stop_cb_wrapper. I think we need
> something similar to what Vignesh has done in one of his bug-fix patch
> [2]. See my comment below in this regard.
You can refer below hunk in 0018.
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
>
> > > >
> > > >
> > > > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > > > ----------------------------------------------------------------------------------
> > > > 1.
> > > > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > > > PG_CATCH();
> > > > {
> > > > /* TODO: Encapsulate cleanup
> > > > from the PG_TRY and PG_CATCH blocks */
> > > > +
> > > > if (iterstate)
> > > > ReorderBufferIterTXNFinish(rb, iterstate);
> > > >
> > > > Spurious line change.
> > > >
> > Done
>
> + /*
> + * We don't expect direct calls to heap_getnext with valid
> + * CheckXidAlive for regular tables. Track that below.
> + */
> + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> + elog(ERROR, "improper heap_getnext call");
>
> Earlier, I thought we don't need to check if it is a regular table in
> this check, but it is required because output plugins can try to do
> that
I did not understand that, can you give some example?
and if they do so during decoding (with historic snapshots), the
> same should be not allowed.
>
> How about changing the error message to "unexpected heap_getnext call
> during logical decoding" or something like that?
Ok
>
> > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > I think that needs to be changed.
> > > >
> > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > -------------------------------------------------------------------------
>
> Few comments on v4-0018-Review-comment-fix-and-refactoring:
> 1.
> + if (streaming)
> + {
> + /*
> + * Set the last last of the stream as the final lsn before calling
> + * stream stop.
> + */
> + txn->final_lsn = prev_lsn;
> + rb->stream_stop(rb, txn);
> + }
>
> Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
Isn't it the same, there we are doing while serializing and here we
are doing while streaming? Basically, the last LSN we streamed. Am I
missing something?
>
> 2.
> + if (streaming)
> + {
> + /*
> + * Set the CheckXidAlive to the current (sub)xid for which this
> + * change belongs to so that we can detect the abort while we are
> + * decoding.
> + */
> + CheckXidAlive = change->txn->xid;
> +
> + /* Increment the stream count. */
> + streamed++;
> + }
>
> Is the variable 'streamed' used anywhere?
>
> 3.
> + /*
> + * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
> + * any memory. We could also keep the hash table and update it with
> + * new ctid values, but this seems simpler and good enough for now.
> + */
> + ReorderBufferDestroyTupleCidHash(rb, txn);
>
> Won't this be required only when we are streaming changes?
I will work on this review comments and reply to them separately along
with the patch.
>
> As per my understanding apart from the above comments, the known
> pending work for this patchset is as follows:
> a. The two open items agreed to you in the email [3].
> b. Complete the handling of schema_sent as discussed above [4].
> c. Few comments by Vignesh and the response on the same by me [5][6].
> d. WAL overhead and performance testing for additional WAL logging by
> this patchset.
> e. Some way to see the tuple for streamed transactions by decoding API
> as speculated by you [7].
>
> Have I missed anything?
I think this is the list I remember. Apart from these few points by
Robert which are still under discussion[8].
>
> [1] - https://www.postgresql.org/message-id/CAA4eK1LOa%2B2KqNX%3Dm%3D1qMBDW%2Bo50AuwjAOX6ZqL-rWGiH1F9MQ%40...
> [2] - https://www.postgresql.org/message-id/CALDaNm3MDxFnsZsnSqVhPBLS3%3DqzNH6%2BYzB%3DxYuX2vbtsUeFgw%40ma...
> [3] - https://www.postgresql.org/message-id/CAFiTN-uT5YZE0egGhKdTteTjcGrPi8hb%3DFMPpr9_hEB7hozQ-Q%40mail.g...
> [4] - https://www.postgresql.org/message-id/CAA4eK1KjD9x0mS4JxzCbu3gu-r6K7XJRV%2BZcGb3BH6U3x2uxew%40mail.g...
> [5] - https://www.postgresql.org/message-id/CALDaNm0DNUojjt7CV-fa59_kFbQQ3rcMBtauvo44ttea7r9KaA%40mail.gma...
> [6] - https://www.postgresql.org/message-id/CAA4eK1%2BZvupW00c--dqEg8f3dHZDOGmA9xOQLyQHjRSoDi6AkQ%40mail.g...
> [7] - https://www.postgresql.org/message-id/CAFiTN-t8PmKA1X4jEqKmkvs0ggWpy0APWpPuaJwpx2YpfAf97w%40mail.gma...
[8] https://www.postgresql.org/message-id/CA%2BTgmoYH6N_YDvKH9AaAJo5ZTHn142K%3DB75VO9yKvjjjHcoZhA%40mail...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-06 08:41 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-06 08:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > It is better to merge it with the main patch for
> > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > difficult to review.
> Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> (0007). Basically, if we merge all of them then we don't need to deal
> with the conflict. I think Tomas has kept them separate so that we
> can review the solution for the schema sent. And, I kept 0018 as a
> separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> In the next patch set, I will merge all of them to 0007.
>
Okay, I think we can merge those patches.
> >
> > + /*
> > + * We don't expect direct calls to heap_getnext with valid
> > + * CheckXidAlive for regular tables. Track that below.
> > + */
> > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> > + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> > + elog(ERROR, "improper heap_getnext call");
> >
> > Earlier, I thought we don't need to check if it is a regular table in
> > this check, but it is required because output plugins can try to do
> > that
> I did not understand that, can you give some example?
>
I think it can lead to the same problem of concurrent aborts as for
catalog scans.
> >
> > > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > > I think that needs to be changed.
> > > > >
> > > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > > -------------------------------------------------------------------------
> >
> > Few comments on v4-0018-Review-comment-fix-and-refactoring:
> > 1.
> > + if (streaming)
> > + {
> > + /*
> > + * Set the last last of the stream as the final lsn before calling
> > + * stream stop.
> > + */
> > + txn->final_lsn = prev_lsn;
> > + rb->stream_stop(rb, txn);
> > + }
> >
> > Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
> Isn't it the same, there we are doing while serializing and here we
> are doing while streaming? Basically, the last LSN we streamed. Am I
> missing something?
>
No, I think you are right.
Few more comments:
--------------------------------
v4-0007-Implement-streaming-mode-in-ReorderBuffer
1.
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about
subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now
= ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn,
command_id);
..
}
Why are we using base snapshot here instead of the snapshot we saved
the first time streaming has happened? And as mentioned in comments,
won't we need to consider the snapshots for subtransactions that
arrived after the last time we have streamed the changes?
2.
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn-
>snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+
txn, command_id);
I don't see where the txn->snapshot_now is getting freed. The
base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
this getting freed.
3.
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+
ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
Is it ever possible that we reach here for subtransaction, if not,
then it should be Assert rather than if condition?
4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
especially to cover the case when it gets called due to memory
overflow (aka via ReorderBufferCheckMemoryLimit).
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin
1.
@@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb,
ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata-
>sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+
}
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn-
>command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+
txn, command_id);
+ rb->stream_stop(rb, txn);
+
+
FlushErrorState();
+ }
Can you update comments either in the above code block or some other
place to explain what is the concurrent abort problem and how we dealt
with it? Also, please explain how the above error handling is
sufficient to address all the various scenarios (sub-transaction got
aborted when we have already sent some changes, or when we have not
sent any changes yet).
v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
1.
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
Why here we can't use TransactionIdDidAbort? If we can't use it, then
can you add comments stating the reason of the same.
2.
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
In comments, there is a mention of a prepared transaction. Do we
allow prepared transactions to be decoded as part of this patch?
3.
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid
(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
This comment just says what code below is doing, can you explain the
rationale behind this check. It would be better if it is clear by
reading comments, why we are doing this check after fetching the
tuple. I think this can refer to the comment I suggested to add for
changes in patch
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-06 10:26 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 10:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > It is better to merge it with the main patch for
> > > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > > difficult to review.
> > Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> > (0007). Basically, if we merge all of them then we don't need to deal
> > with the conflict. I think Tomas has kept them separate so that we
> > can review the solution for the schema sent. And, I kept 0018 as a
> > separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> > In the next patch set, I will merge all of them to 0007.
> >
>
> Okay, I think we can merge those patches.
ok
>
> > >
> > > + /*
> > > + * We don't expect direct calls to heap_getnext with valid
> > > + * CheckXidAlive for regular tables. Track that below.
> > > + */
> > > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > > + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> > > + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> > > + elog(ERROR, "improper heap_getnext call");
> > >
> > > Earlier, I thought we don't need to check if it is a regular table in
> > > this check, but it is required because output plugins can try to do
> > > that
> > I did not understand that, can you give some example?
> >
>
> I think it can lead to the same problem of concurrent aborts as for
> catalog scans.
Yeah, got it.
>
> > >
> > > > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > > > I think that needs to be changed.
> > > > > >
> > > > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > > > -------------------------------------------------------------------------
> > >
> > > Few comments on v4-0018-Review-comment-fix-and-refactoring:
> > > 1.
> > > + if (streaming)
> > > + {
> > > + /*
> > > + * Set the last last of the stream as the final lsn before calling
> > > + * stream stop.
> > > + */
> > > + txn->final_lsn = prev_lsn;
> > > + rb->stream_stop(rb, txn);
> > > + }
> > >
> > > Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
> > Isn't it the same, there we are doing while serializing and here we
> > are doing while streaming? Basically, the last LSN we streamed. Am I
> > missing something?
> >
>
> No, I think you are right.
>
> Few more comments:
> --------------------------------
> v4-0007-Implement-streaming-mode-in-ReorderBuffer
> 1.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about
> subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now
> = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn,
> command_id);
> ..
> }
>
> Why are we using base snapshot here instead of the snapshot we saved
> the first time streaming has happened? And as mentioned in comments,
> won't we need to consider the snapshots for subtransactions that
> arrived after the last time we have streamed the changes?
>
> 2.
> + /* remember the command ID and snapshot for the streaming run */
> + txn->command_id = command_id;
> + txn-
> >snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
>
> I don't see where the txn->snapshot_now is getting freed. The
> base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
> this getting freed.
Ok, I will check that and fix.
>
> 3.
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * If this is a subxact, we need to stream the top-level transaction
> + * instead.
> + */
> + if (txn->toptxn)
> + {
> +
> ReorderBufferStreamTXN(rb, txn->toptxn);
> + return;
> + }
>
> Is it ever possible that we reach here for subtransaction, if not,
> then it should be Assert rather than if condition?
ReorderBufferCheckMemoryLimit, can call it either for the
subtransaction or for the main transaction, depends upon in which
ReorderBufferTXN you are adding the current change.
I will analyze your other comments and fix them in the next version.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-06 11:06 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-06 11:06 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3.
> > +static void
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > + /*
> > + * If this is a subxact, we need to stream the top-level transaction
> > + * instead.
> > + */
> > + if (txn->toptxn)
> > + {
> > +
> > ReorderBufferStreamTXN(rb, txn->toptxn);
> > + return;
> > + }
> >
> > Is it ever possible that we reach here for subtransaction, if not,
> > then it should be Assert rather than if condition?
>
> ReorderBufferCheckMemoryLimit, can call it either for the
> subtransaction or for the main transaction, depends upon in which
> ReorderBufferTXN you are adding the current change.
>
That function has code like below:
ReorderBufferCheckMemoryLimit()
{
..
if (ReorderBufferCanStream(rb))
{
/*
* Pick the largest toplevel transaction and evict it from memory by
* streaming the already decoded part.
*/
txn = ReorderBufferLargestTopTXN(rb);
/* we know there has to be one, because the size is not zero */
Assert(txn && !txn->toptxn);
..
ReorderBufferStreamTXN(rb, txn);
..
}
How can it ReorderBufferTXN pass for subtransaction?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-06 11:14 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 11:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 4:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > 3.
> > > +static void
> > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > {
> > > ..
> > > + /*
> > > + * If this is a subxact, we need to stream the top-level transaction
> > > + * instead.
> > > + */
> > > + if (txn->toptxn)
> > > + {
> > > +
> > > ReorderBufferStreamTXN(rb, txn->toptxn);
> > > + return;
> > > + }
> > >
> > > Is it ever possible that we reach here for subtransaction, if not,
> > > then it should be Assert rather than if condition?
> >
> > ReorderBufferCheckMemoryLimit, can call it either for the
> > subtransaction or for the main transaction, depends upon in which
> > ReorderBufferTXN you are adding the current change.
> >
>
> That function has code like below:
>
> ReorderBufferCheckMemoryLimit()
> {
> ..
> if (ReorderBufferCanStream(rb))
> {
> /*
> * Pick the largest toplevel transaction and evict it from memory by
> * streaming the already decoded part.
> */
> txn = ReorderBufferLargestTopTXN(rb);
> /* we know there has to be one, because the size is not zero */
> Assert(txn && !txn->toptxn);
> ..
> ReorderBufferStreamTXN(rb, txn);
> ..
> }
>
> How can it ReorderBufferTXN pass for subtransaction?
>
Hmm, I missed it. You are right, will fix it.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-08 07:42 Dilip Kumar <dilipbalaut@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-08 07:42 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 4:44 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 4:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > 3.
> > > > +static void
> > > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > > {
> > > > ..
> > > > + /*
> > > > + * If this is a subxact, we need to stream the top-level transaction
> > > > + * instead.
> > > > + */
> > > > + if (txn->toptxn)
> > > > + {
> > > > +
> > > > ReorderBufferStreamTXN(rb, txn->toptxn);
> > > > + return;
> > > > + }
> > > >
> > > > Is it ever possible that we reach here for subtransaction, if not,
> > > > then it should be Assert rather than if condition?
> > >
> > > ReorderBufferCheckMemoryLimit, can call it either for the
> > > subtransaction or for the main transaction, depends upon in which
> > > ReorderBufferTXN you are adding the current change.
> > >
> >
> > That function has code like below:
> >
> > ReorderBufferCheckMemoryLimit()
> > {
> > ..
> > if (ReorderBufferCanStream(rb))
> > {
> > /*
> > * Pick the largest toplevel transaction and evict it from memory by
> > * streaming the already decoded part.
> > */
> > txn = ReorderBufferLargestTopTXN(rb);
> > /* we know there has to be one, because the size is not zero */
> > Assert(txn && !txn->toptxn);
> > ..
> > ReorderBufferStreamTXN(rb, txn);
> > ..
> > }
> >
> > How can it ReorderBufferTXN pass for subtransaction?
> >
> Hmm, I missed it. You are right, will fix it.
>
I have observed one more design issue. The problem is that when we
get a toasted chunks we remember the changes in the memory(hash table)
but don't stream until we get the actual change on the main table.
Now, the problem is that we might get the change of the toasted table
and the main table in different streams. So basically, in a stream,
if we have only got the toasted tuples then even after
ReorderBufferStreamTXN the memory usage will not be reduced.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-09 04:05 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-09 04:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have observed one more design issue.
>
Good observation.
> The problem is that when we
> get a toasted chunks we remember the changes in the memory(hash table)
> but don't stream until we get the actual change on the main table.
> Now, the problem is that we might get the change of the toasted table
> and the main table in different streams. So basically, in a stream,
> if we have only got the toasted tuples then even after
> ReorderBufferStreamTXN the memory usage will not be reduced.
>
I think we can't split such changes in a different stream (unless we
design an entirely new solution to send partial changes of toast
data), so we need to send them together. We can keep a flag like
data_complete in ReorderBufferTxn and mark it complete only when we
are able to assemble the entire tuple. Now, whenever, we try to
stream the changes once we reach the memory threshold, we can check
whether the data_complete flag is true, if so, then only send the
changes, otherwise, we can pick the next largest transaction. I think
we can retry it for few times and if we get the incomplete data for
multiple transactions, then we can decide to spill the transaction or
maybe we can directly spill the first largest transaction which has
incomplete data.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-09 05:00 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-09 05:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have observed one more design issue.
> >
>
> Good observation.
>
> > The problem is that when we
> > get a toasted chunks we remember the changes in the memory(hash table)
> > but don't stream until we get the actual change on the main table.
> > Now, the problem is that we might get the change of the toasted table
> > and the main table in different streams. So basically, in a stream,
> > if we have only got the toasted tuples then even after
> > ReorderBufferStreamTXN the memory usage will not be reduced.
> >
>
> I think we can't split such changes in a different stream (unless we
> design an entirely new solution to send partial changes of toast
> data), so we need to send them together. We can keep a flag like
> data_complete in ReorderBufferTxn and mark it complete only when we
> are able to assemble the entire tuple. Now, whenever, we try to
> stream the changes once we reach the memory threshold, we can check
> whether the data_complete flag is true, if so, then only send the
> changes, otherwise, we can pick the next largest transaction. I think
> we can retry it for few times and if we get the incomplete data for
> multiple transactions, then we can decide to spill the transaction or
> maybe we can directly spill the first largest transaction which has
> incomplete data.
>
Yeah, we might do something on this line. Basically, we need to mark
the top-transaction as data-incomplete if any of its subtransaction is
having data-incomplete (it will always be the latest sub-transaction
of the top transaction). Also, for streaming, we are checking the
largest top transaction whereas for spilling we just need the larget
(sub) transaction. So we also need to decide while picking the
largest top transaction for streaming, if we get a few transactions
with in-complete data then how we will go for the spill. Do we spill
all the sub-transactions under this top transaction or we will again
find the larget (sub) transaction for spilling.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-09 06:39 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-09 06:39 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have observed one more design issue.
> > >
> >
> > Good observation.
> >
> > > The problem is that when we
> > > get a toasted chunks we remember the changes in the memory(hash table)
> > > but don't stream until we get the actual change on the main table.
> > > Now, the problem is that we might get the change of the toasted table
> > > and the main table in different streams. So basically, in a stream,
> > > if we have only got the toasted tuples then even after
> > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > >
> >
> > I think we can't split such changes in a different stream (unless we
> > design an entirely new solution to send partial changes of toast
> > data), so we need to send them together. We can keep a flag like
> > data_complete in ReorderBufferTxn and mark it complete only when we
> > are able to assemble the entire tuple. Now, whenever, we try to
> > stream the changes once we reach the memory threshold, we can check
> > whether the data_complete flag is true, if so, then only send the
> > changes, otherwise, we can pick the next largest transaction. I think
> > we can retry it for few times and if we get the incomplete data for
> > multiple transactions, then we can decide to spill the transaction or
> > maybe we can directly spill the first largest transaction which has
> > incomplete data.
> >
> Yeah, we might do something on this line. Basically, we need to mark
> the top-transaction as data-incomplete if any of its subtransaction is
> having data-incomplete (it will always be the latest sub-transaction
> of the top transaction). Also, for streaming, we are checking the
> largest top transaction whereas for spilling we just need the larget
> (sub) transaction. So we also need to decide while picking the
> largest top transaction for streaming, if we get a few transactions
> with in-complete data then how we will go for the spill. Do we spill
> all the sub-transactions under this top transaction or we will again
> find the larget (sub) transaction for spilling.
>
I think it is better to do later as that will lead to the spill of
only required (minimum changes to get the memory below threshold)
changes.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-09 07:10 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-09 07:10 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have observed one more design issue.
> > > >
> > >
> > > Good observation.
> > >
> > > > The problem is that when we
> > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > but don't stream until we get the actual change on the main table.
> > > > Now, the problem is that we might get the change of the toasted table
> > > > and the main table in different streams. So basically, in a stream,
> > > > if we have only got the toasted tuples then even after
> > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > >
> > >
> > > I think we can't split such changes in a different stream (unless we
> > > design an entirely new solution to send partial changes of toast
> > > data), so we need to send them together. We can keep a flag like
> > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > are able to assemble the entire tuple. Now, whenever, we try to
> > > stream the changes once we reach the memory threshold, we can check
> > > whether the data_complete flag is true, if so, then only send the
> > > changes, otherwise, we can pick the next largest transaction. I think
> > > we can retry it for few times and if we get the incomplete data for
> > > multiple transactions, then we can decide to spill the transaction or
> > > maybe we can directly spill the first largest transaction which has
> > > incomplete data.
> > >
> > Yeah, we might do something on this line. Basically, we need to mark
> > the top-transaction as data-incomplete if any of its subtransaction is
> > having data-incomplete (it will always be the latest sub-transaction
> > of the top transaction). Also, for streaming, we are checking the
> > largest top transaction whereas for spilling we just need the larget
> > (sub) transaction. So we also need to decide while picking the
> > largest top transaction for streaming, if we get a few transactions
> > with in-complete data then how we will go for the spill. Do we spill
> > all the sub-transactions under this top transaction or we will again
> > find the larget (sub) transaction for spilling.
> >
>
> I think it is better to do later as that will lead to the spill of
> only required (minimum changes to get the memory below threshold)
> changes.
Make sense to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 04:44 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-10 04:44 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > It is better to merge it with the main patch for
> > > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > > difficult to review.
> > Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> > (0007). Basically, if we merge all of them then we don't need to deal
> > with the conflict. I think Tomas has kept them separate so that we
> > can review the solution for the schema sent. And, I kept 0018 as a
> > separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> > In the next patch set, I will merge all of them to 0007.
> >
>
> Okay, I think we can merge those patches.
Done
0008, 0009, 0017, 0018 are merged to 0007, 0012 is merged to 0010
>
> Few more comments:
> --------------------------------
> v4-0007-Implement-streaming-mode-in-ReorderBuffer
> 1.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about
> subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now
> = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn,
> command_id);
> ..
> }
>
> Why are we using base snapshot here instead of the snapshot we saved
> the first time streaming has happened? And as mentioned in comments,
> won't we need to consider the snapshots for subtransactions that
> arrived after the last time we have streamed the changes?
Fixed
>
> 2.
> + /* remember the command ID and snapshot for the streaming run */
> + txn->command_id = command_id;
> + txn-
> >snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
>
> I don't see where the txn->snapshot_now is getting freed. The
> base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
> this getting freed.
I have freed this In ReorderBufferCleanupTXN
>
> 3.
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * If this is a subxact, we need to stream the top-level transaction
> + * instead.
> + */
> + if (txn->toptxn)
> + {
> +
> ReorderBufferStreamTXN(rb, txn->toptxn);
> + return;
> + }
>
> Is it ever possible that we reach here for subtransaction, if not,
> then it should be Assert rather than if condition?
Fixed
>
> 4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
> fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
> especially to cover the case when it gets called due to memory
> overflow (aka via ReorderBufferCheckMemoryLimit).
We get origin_lsn during commit time so I am not sure how can we do
that. I have also noticed that currently, we are not using origin_lsn
on the subscriber side. I think need more investigation that if we
want this then do we need to log it early.
>
> v4-0017-Extend-handling-of-concurrent-aborts-for-streamin
> 1.
> @@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb,
> ReorderBufferTXN *txn)
> if (using_subtxn)
>
> RollbackAndReleaseCurrentSubTransaction();
>
> - PG_RE_THROW();
> + /* re-throw only if it's not an abort */
> + if (errdata-
> >sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
> + {
> + MemoryContextSwitchTo(ecxt);
> + PG_RE_THROW();
> +
> }
> + else
> + {
> + /* remember the command ID and snapshot for the streaming run */
> + txn-
> >command_id = command_id;
> + txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
> + rb->stream_stop(rb, txn);
> +
> +
> FlushErrorState();
> + }
>
> Can you update comments either in the above code block or some other
> place to explain what is the concurrent abort problem and how we dealt
> with it? Also, please explain how the above error handling is
> sufficient to address all the various scenarios (sub-transaction got
> aborted when we have already sent some changes, or when we have not
> sent any changes yet).
Done
>
> v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
> 1.
> + /*
> + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> + * error out
> + */
> + if (TransactionIdIsValid(CheckXidAlive) &&
> + !TransactionIdIsInProgress(CheckXidAlive) &&
> + !TransactionIdDidCommit(CheckXidAlive))
> + ereport(ERROR,
> + (errcode(ERRCODE_TRANSACTION_ROLLBACK),
> + errmsg("transaction aborted during system catalog scan")));
>
> Why here we can't use TransactionIdDidAbort? If we can't use it, then
> can you add comments stating the reason of the same.
Done
>
> 2.
> /*
> + * An xid value pointing to a possibly ongoing or a prepared transaction.
> + * Currently used in logical decoding. It's possible that such transactions
> + * can get aborted while the decoding is ongoing.
> + */
> +TransactionId CheckXidAlive = InvalidTransactionId;
>
> In comments, there is a mention of a prepared transaction. Do we
> allow prepared transactions to be decoded as part of this patch?
Fixed
>
> 3.
> + /*
> + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> + * error out
> + */
> + if (TransactionIdIsValid
> (CheckXidAlive) &&
> + !TransactionIdIsInProgress(CheckXidAlive) &&
> + !TransactionIdDidCommit(CheckXidAlive))
>
> This comment just says what code below is doing, can you explain the
> rationale behind this check. It would be better if it is clear by
> reading comments, why we are doing this check after fetching the
> tuple. I think this can refer to the comment I suggested to add for
> changes in patch
> v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.
Done
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v5-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 9924675dc35109c8823ee5a748e293d6917de1b7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v5 01/14] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b..51557e2 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa6812..8c281e8 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4..4805cae 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v5-0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 2c2dbdf2ea938c8115a88c83255a77ecac371bec Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v5 04/14] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d9..a3efbfd 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389..9c95fc1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d16bebf..9e84687 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v5-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 16.4K) download
From d35181a5128caaa300131f5a9674d15614cc0f57 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v5 02/14] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 235 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc7..6e46d19 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2..dd3d36f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f352805..723300b 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2227,6 +2234,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2674,6 +2713,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2770,6 +2810,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3055,6 +3096,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e42d4c5..d16bebf 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v5-0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From 3c783cce62294b92c51d9f4dba316443f25e7cee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v5 05/14] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 652a76e..9eda992 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -741,7 +741,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -761,7 +761,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -784,7 +784,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -844,7 +844,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -860,7 +860,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1081,7 +1081,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1110,7 +1110,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1274,7 +1274,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1322,7 +1322,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1337,7 +1337,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1354,7 +1354,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1987,7 +1987,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2135,7 +2135,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2314,7 +2314,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2331,7 +2331,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2351,7 +2351,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2539,12 +2539,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 9e84687..04dd0cb 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v5-0003-fixup-is_schema_sent-set-too-early.patch (application/octet-stream, 858B) download
From bd1088604c7917441322bcdac6af6c45c4cf765c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH v5 03/14] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 723300b..652a76e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1837,7 +1837,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v5-0006-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 13.1K) download
From 74663de44159dfe79a6d6dca3f1d8b7d0453a886 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v5 06/14] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7b8490d..2d4ef48 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5644b8d 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9eda992..c9d28a3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v5-0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.2K) download
From 60d03b120ef3ca7a62be67d78294322fb9dd5341 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 6 Jan 2020 13:15:24 +0530
Subject: [PATCH v5 07/14] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 691 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 32 ++
3 files changed, 668 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c9d28a3..c381d8d 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,40 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2094,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2128,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2262,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2030,6 +2353,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2165,8 +2495,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2174,6 +2513,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2185,19 +2525,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2226,6 +2575,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2315,6 +2665,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2419,6 +2776,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2438,15 +2827,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2739,6 +3159,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 04dd0cb..c4a2643 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v5-0009-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 70b890b5e730d7c08661c3bfd6eecf149d766f2d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v5 09/14] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c902..8b1923c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..5cd1daa 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 95962b4..c45c2ce 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 42e3e04..16f9d00 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..48b960c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..536722b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..3394379 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index a276237..66e89f0 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v5-0008-Fix-speculative-insert-bug.patch (application/octet-stream, 2.5K) download
From 490495a4cd8ac57753f9977122bbd9887f4c09c1 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v5 08/14] Fix speculative insert bug.
---
src/backend/replication/logical/reorderbuffer.c | 23 +++++++++++++++++++----
src/include/replication/reorderbuffer.h | 6 ++++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c381d8d..eb6fda5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c4a2643..f41e216 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -335,6 +335,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
1.8.3.1
v5-0010-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 91.3K) download
From 8af095c64b08a228fdfdae2e6fdee5d3b686b9f2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v5 10/14] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa..1dc486c 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c45c2ce..1ece10d 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486b..03ef76c 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 16f9d00..61701d0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..e80d00c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1..61064f3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index dcf7c08..918a841 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c..5c20c0e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b..ebe0423 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +652,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -549,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -623,6 +875,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683..3085c0f 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c06374..63fc2c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379..18f416f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e5a5d02..bbc9112 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..ade4188 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 66e89f0..1e4269c 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v5-0011-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 6f2811a82d859b692b6b1a8e59083c6ab5ac953b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v5 11/14] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb5811..180ea88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 773edf8..cb9e6ee 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index eb6fda5..9dd379e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3264,6 +3268,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7..d7f22ae 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 427faa3..9ef4fbf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f41e216..d671a2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62eaf90..2dcb063 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1960,9 +1960,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v5-0013-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 657b9d3dd5df36dc04aa8138b071bb8eb3165024 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v5 13/14] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9dd379e..5e8a931 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v5-0014-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From fd866d4e41a1fe3d61606029b529998eb5403c14 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v5 14/14] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v5-0012-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From c01738e8cc2480efbc13af94b050ff83607eb589 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v5 12/14] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 04:51 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-10 04:51 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 30, 2019 at 3:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > 0002-Issue-individual-invalidations-with-wal_level-log
> > > > ----------------------------------------------------------------------------
> > > > 1.
> > > > xact_desc_invalidations(StringInfo buf,
> > > > {
> > > > ..
> > > > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > > > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> > > >
> > > > You have removed logging for the above cache but forgot to remove its
> > > > reference from one of the places. Also, I think you need to add a
> > > > comment somewhere in inval.c to say why you are writing for WAL for
> > > > some types of invalidations and not for others?
> > Done
> >
>
> I don't see any new comments as asked by me.
Done
I think we should also
> consider WAL logging at each command end instead of doing piecemeal as
> discussed in another email [1], which will have lesser code changes
> and maybe better in performance. You might want to evaluate the
> performance of both approaches.
Still pending, will work on this.
>
> > > > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > > > ----------------------------------------------------------------------------------
> > > > 1.
> > > > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > > > PG_CATCH();
> > > > {
> > > > /* TODO: Encapsulate cleanup
> > > > from the PG_TRY and PG_CATCH blocks */
> > > > +
> > > > if (iterstate)
> > > > ReorderBufferIterTXNFinish(rb, iterstate);
> > > >
> > > > Spurious line change.
> > > >
> > Done
>
> + /*
> + * We don't expect direct calls to heap_getnext with valid
> + * CheckXidAlive for regular tables. Track that below.
> + */
> + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> + elog(ERROR, "improper heap_getnext call");
>
> Earlier, I thought we don't need to check if it is a regular table in
> this check, but it is required because output plugins can try to do
> that and if they do so during decoding (with historic snapshots), the
> same should be not allowed.
>
> How about changing the error message to "unexpected heap_getnext call
> during logical decoding" or something like that?
Done
>
> > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > I think that needs to be changed.
> > > >
> > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > -------------------------------------------------------------------------
>
> Few comments on v4-0018-Review-comment-fix-and-refactoring:
> 1.
> + if (streaming)
> + {
> + /*
> + * Set the last last of the stream as the final lsn before calling
> + * stream stop.
> + */
> + txn->final_lsn = prev_lsn;
> + rb->stream_stop(rb, txn);
> + }
>
> Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
Already agreed upon current implementation
>
> 2.
> + if (streaming)
> + {
> + /*
> + * Set the CheckXidAlive to the current (sub)xid for which this
> + * change belongs to so that we can detect the abort while we are
> + * decoding.
> + */
> + CheckXidAlive = change->txn->xid;
> +
> + /* Increment the stream count. */
> + streamed++;
> + }
>
> Is the variable 'streamed' used anywhere?
Removed
>
> 3.
> + /*
> + * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
> + * any memory. We could also keep the hash table and update it with
> + * new ctid values, but this seems simpler and good enough for now.
> + */
> + ReorderBufferDestroyTupleCidHash(rb, txn);
>
> Won't this be required only when we are streaming changes?
Fixed
>
> As per my understanding apart from the above comments, the known
> pending work for this patchset is as follows:
> a. The two open items agreed to you in the email [3].
> b. Complete the handling of schema_sent as discussed above [4].
> c. Few comments by Vignesh and the response on the same by me [5][6].
> d. WAL overhead and performance testing for additional WAL logging by
> this patchset.
> e. Some way to see the tuple for streamed transactions by decoding API
> as speculated by you [7].
>
> Have I missed anything?
I have worked upon most of these items, I will reply to them separately.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 05:22 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-10 05:22 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 30, 2019 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Dec 29, 2019 at 1:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have observed some more issues
> >
> > 1. Currently, In ReorderBufferCommit, it is always expected that
> > whenever we get REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM, we must
> > have already got REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT and in
> > SPEC_CONFIRM we send the tuple we got in SPECT_INSERT. But, now those
> > two messages can be in different streams. So we need to find a way to
> > handle this. Maybe once we get SPEC_INSERT then we can remember the
> > tuple and then if we get the SPECT_CONFIRM in the next stream we can
> > send that tuple?
> >
>
> Your suggestion makes sense to me. So, we can try it.
I have implemented this and attached it as a separate patch. In my
latest patch set[1]
>
> > 2. During commit time in DecodeCommit we check whether we need to skip
> > the changes of the transaction or not by calling
> > SnapBuildXactNeedsSkip but since now we support streaming so it's
> > possible that before we decode the commit WAL, we might have already
> > sent the changes to the output plugin even though we could have
> > skipped those changes. So my question is instead of checking at the
> > commit time can't we check before adding to ReorderBuffer itself
> >
>
> I think if we can do that then the same will be true for current code
> irrespective of this patch. I think it is possible that we can't take
> that decision while decoding because we haven't assembled a consistent
> snapshot yet. I think we might be able to do that while we try to
> stream the changes. I think we need to take care of all the
> conditions during streaming (when the logical_decoding_workmem limit
> is reached) as we do in DecodeCommit. This needs a bit more study.
I have analyzed this further and I think we can not decide all the
conditions even while streaming. Because IMHO once we get the
SNAPBUILD_FULL_SNAPSHOT we can add the changes to the reorder buffer
so that if we get the commit of the transaction after we reach to the
SNAPBUILD_CONSISTENT. However, if we get the commit before we reach
to SNAPBUILD_CONSISTENT then we need to ignore this transaction. Now,
even if we have SNAPBUILD_FULL_SNAPSHOT we can stream the changes
which might get dropped later but that we can not decide while
streaming.
[1] https://www.postgresql.org/message-id/CAFiTN-snMb%3D53oqkM8av8Lqfxojjm4OBwCNxmFssgLCceY_zgg%40mail.g...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 20:56 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-10 20:56 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
I pushed 0005 (the rbtxn flags thing) after some light editing.
It's been around for long enough ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 21:35 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-10 21:35 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Here's a rebase of this patch series. I didn't change anything except
1. disregard what was 0005, since I already pushed it.
2. roll 0003 into 0002.
3. rebase 0007 (now 0005) to account for the reorderbuffer changes.
(I did notice that 0005 adds a new boolean any_data_sent, which is
silly -- it should be another txn_flags bit.)
However, tests don't pass for me; notably, test_decoding crashes.
OTOH I noticed that the streamed transaction support in test_decoding
writes the XID to the output, which is going to make it useless for
regression testing. It probably should not emit the numerical values.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-10 21:37 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-10 21:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-Jan-10, Alvaro Herrera wrote:
> Here's a rebase of this patch series. I didn't change anything except
... this time with attachments ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v6-0001-Immediately-WAL-log-assignments.patch (text/x-diff, 10.4K) download
From f2cd3a14f9513b880becedd95c933dad53c7d9e3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v6 01/12] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++--
src/backend/access/transam/xlogreader.c | 5 +++
src/backend/replication/logical/decode.c | 39 ++++++++++----------
src/include/access/xact.h | 3 ++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 ++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b6d8..51557e2951 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f667..b11b0c2940 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa68127a3..8c281e821a 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a651..a99fcaf0a1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04babc2..8645b3816c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033fc20..e23892ab87 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4e1d..4805caec67 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0194..2f0c8bf589 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
2.20.1
v6-0002-Issue-individual-invalidations-with-wal_level-log.patch (text/x-diff, 16.3K) download
From ccc7defbeec7653240c76e6e4dcdb9349b49284d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v6 02/12] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 23 ++++++
.../replication/logical/reorderbuffer.c | 55 ++++++++++++--
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++
src/include/access/xact.h | 18 ++++-
src/include/replication/reorderbuffer.h | 14 ++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc714a..6e46d19168 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2951..dd3d36ffb1 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf0a1..13a11ac782 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bbd908af05..531897cf05 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2225,6 +2231,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2674,6 +2712,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2770,6 +2809,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3055,6 +3095,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33be6..e0d04b9850 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b3816c..6f2a5831ee 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -197,6 +197,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
+/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4f6c65d6f4..fa41115db9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -458,6 +469,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
2.20.1
v6-0003-Extend-the-output-plugin-API-with-stream-methods.patch (text/x-diff, 34.8K) download
From 1236dc75d5609f14d359afb66a0a5f8bbfbf353e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v6 03/12] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 ++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 +++++
src/include/replication/reorderbuffer.h | 57 ++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d91e0..a3efbfdd76 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db968641e..ace21ec8e5 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389a57..9c95fc1ed8 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f1da..f24e2468ac 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -79,6 +79,11 @@ typedef struct LogicalDecodingContext
*/
void *output_writer_private;
+ /*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
/*
* State for writing output.
*/
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236c57..0d0a94a648 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -99,6 +99,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
+/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
/*
* Output plugin callbacks
*/
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index fa41115db9..79ea33cd26 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -355,6 +355,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
+ /*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
/*
* Pointer that will be passed untouched to the callbacks.
*/
--
2.20.1
v6-0004-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (text/x-diff, 13.0K) download
From 06973f7b57a9c186e53400e8815d8edf7a6bc047 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v6 04/12] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 +-
src/backend/access/heap/heapam.c | 41 ++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++
.../replication/logical/reorderbuffer.c | 8 +--
src/backend/utils/time/snapmgr.c | 25 +++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec8e5..319349a92d 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7b8490d4e5..2d4ef48069 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1421,6 +1430,14 @@ heap_fetch(Relation relation,
OffsetNumber offnum;
bool valid;
+ /*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
/*
* Fetch and pin the appropriate page of the relation.
*/
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1682,6 +1707,14 @@ heap_get_latest_tid(TableScanDesc sscan,
*/
Assert(ItemPointerIsValid(tid));
+ /*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
/*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05416..5644b8d41a 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 531897cf05..2da0a23a7e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c592c..93a0c048c5 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -153,6 +153,13 @@ static Snapshot SecondarySnapshot = NULL;
static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
+/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13ce84..12f737b21a 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
2.20.1
v6-0005-Implement-streaming-mode-in-ReorderBuffer.patch (text/x-diff, 37.0K) download
From 84b94c7c49ca7aae737ffb1451eee9098c483578 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v6 05/12] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
.../replication/logical/reorderbuffer.c | 691 +++++++++++++++---
src/include/replication/reorderbuffer.h | 36 +
3 files changed, 672 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba10890aa..160b167adb 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2da0a23a7e..50341a6d9e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -768,6 +781,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1319,6 +1373,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_delete(&txn->base_snapshot_node);
}
+ /*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
/*
* Remove TXN from its containing list.
*
@@ -1344,9 +1407,94 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferReturnTXN(rb, txn);
}
+/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferIterTXNState *volatile iterstate = NULL;
-
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
-
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
-
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,40 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2094,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,17 +2128,116 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
- PG_RE_THROW();
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
+/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
@@ -1946,6 +2262,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2030,6 +2353,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2165,8 +2495,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2174,6 +2513,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2185,19 +2525,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2226,6 +2575,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2315,6 +2665,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2418,6 +2775,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
return largest;
}
+/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
@@ -2438,15 +2827,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2739,6 +3159,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 79ea33cd26..629eeca7f6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
/*
* LSN pointing to the end of the commit record + 1.
*/
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
XLogRecPtr base_snapshot_lsn;
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
+ /*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
/*
* How many ReorderBufferChange's do we have in this txn.
*
--
2.20.1
v6-0006-Fix-speculative-insert-bug.patch (text/x-diff, 2.5K) download
From ae7bfc6848143185d13adaa5532c13e9f3d730ca Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v6 06/12] Fix speculative insert bug.
---
.../replication/logical/reorderbuffer.c | 23 +++++++++++++++----
src/include/replication/reorderbuffer.h | 6 +++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 50341a6d9e..8e4744f73a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 629eeca7f6..0510d3831f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -343,6 +343,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
2.20.1
v6-0007-Support-logical_decoding_work_mem-set-from-create.patch (text/x-diff, 13.1K) download
From c00cf7ddb5f49d355c52b704479373ac6598134b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v6 07/12] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 ++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c90282f..8b1923c9de 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..91790b0c95 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83bd2e..5cd1daa238 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 95962b4a3e..c45c2ce212 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 42e3e04e68..16f9d008fd 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f95c..48b960c4c9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 752508213a..536722b32f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..3394379f86 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index a276237477..66e89f087b 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1
v6-0008-Add-support-for-streaming-to-built-in-replication.patch (text/x-diff, 91.3K) download
From 934dcdd48e4720cb558e6e7a1049f80b7e218344 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v6 08/12] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 +++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../t/012_stream_subxact_abort.pl | 82 ++
.../t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..e1fb9075e1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0c95..d9abf5e64c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa238..1dc486c0e7 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c45c2ce212..1ece10d9f5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486bebd..03ef76caea 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 16f9d008fd..61701d0590 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e987..e80d00c1c3 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1ed8..61064f392a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index dcf7c08c18..918a841125 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c4c9..5c20c0e4c3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
errmsg("ORIGIN message sent out of order")));
}
+/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
/*
* Handle RELATION message.
*
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1116,6 +1569,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
}
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
/*
* Apply main loop.
*/
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b32f..ebe0423cc2 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -512,6 +651,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
rel_sync_cache_publication_cb(arg, cacheid, hashvalue);
}
+/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
/*
* Initialize the relation schema sync cache for a decoding session.
*
@@ -548,6 +772,34 @@ init_rel_sync_cache(MemoryContext cachectx)
(Datum) 0);
}
+/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
/*
* Find or create entry in the relation schema cache.
*/
@@ -622,6 +874,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
/*
* Relcache invalidation callback
*/
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683e2a..3085c0f921 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c063749b6..63fc2c7ff2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndPrepareWrite, WalSndWriteData,
WalSndUpdateProgress);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
/*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379f86..18f416fa78 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e5a5d025ba..bbc9112b01 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4db3..ade4188dd2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 66e89f087b..1e4269ca21 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000000..2f01133f69
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000000..d2ae38592b
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000000..0da39a1a8a
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000000..402df30f59
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000000..becbdd0578
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.20.1
v6-0009-Track-statistics-for-streaming.patch (text/x-diff, 11.7K) download
From 31d5e1ed8ced85ea01de18fbd70af0ad38956e55 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v6 09/12] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
.../replication/logical/reorderbuffer.c | 13 ++++++++
src/backend/replication/walsender.c | 32 ++++++++++++++++---
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 13 +++++---
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 ++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb58115af..180ea880a4 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 773edf85e7..cb9e6ee9ea 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8e4744f73a..16515d0a12 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3264,6 +3268,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7ff2..d7f22ae960 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 427faa3c3b..9ef4fbf4f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0510d3831f..7259c66c3f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -524,15 +524,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f0a4..3888b0c2f8 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62eaf90a0f..2dcb063912 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1960,9 +1960,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
2.20.1
v6-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (text/x-diff, 14.8K) download
From 7d671806584fff71067c8bde38b2f642ba1331a9 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v6 10/12] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560b23..8cd1993393 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2fbc..94c71f8ae2 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e7fe..ad3ed13ffc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9181..a6fae9c3f1 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17a78..202871a658 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10a19..70c86b22ac 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6d63..f9c8d1d348 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7332..0c9c6b3dd4 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bdba9..21f50c7012 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133f69..30561d8f96 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae38592b..9a6bac6822 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bdc35..ed56fbf96c 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cba4c..4df1ddef63 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1a8a..c3caff6149 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7c2f..c62eb521e7 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30f59..2be7542831 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0578..2da9607a7d 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9435..96ffc091b0 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
2.20.1
v6-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (text/x-diff, 1016B) download
From ab09b5f50aa9af74da48148818b1899f907038eb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v6 11/12] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 16515d0a12..c0c6a7f86c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
2.20.1
v6-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (text/x-diff, 4.4K) download
From b9bb007d0dccb301acac5c0da13bd1d41e38428a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v6 12/12] Add TAP test for streaming vs. DDL
---
.../subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000000..b8d78b1972
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.20.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-11 02:48 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-11 02:48 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-Jan-10, Alvaro Herrera wrote:
> From 7d671806584fff71067c8bde38b2f642ba1331a9 Mon Sep 17 00:00:00 2001
> From: Dilip Kumar <dilip.kumar@enterprisedb.com>
> Date: Wed, 20 Nov 2019 16:41:13 +0530
> Subject: [PATCH v6 10/12] Enable streaming for all subscription TAP tests
This patch turns a lot of test into the streamed mode. While it's
great that streaming mode is tested, we should add new tests for it
rather than failing to keep tests for the non-streamed mode. I suggest
that we add two versions of each test, one for each mode. Maybe the way
to do that is to create some subroutine that can be called twice.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-13 09:48 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-13 09:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have observed one more design issue.
> > > >
> > >
> > > Good observation.
> > >
> > > > The problem is that when we
> > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > but don't stream until we get the actual change on the main table.
> > > > Now, the problem is that we might get the change of the toasted table
> > > > and the main table in different streams. So basically, in a stream,
> > > > if we have only got the toasted tuples then even after
> > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > >
> > >
> > > I think we can't split such changes in a different stream (unless we
> > > design an entirely new solution to send partial changes of toast
> > > data), so we need to send them together. We can keep a flag like
> > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > are able to assemble the entire tuple. Now, whenever, we try to
> > > stream the changes once we reach the memory threshold, we can check
> > > whether the data_complete flag is true, if so, then only send the
> > > changes, otherwise, we can pick the next largest transaction. I think
> > > we can retry it for few times and if we get the incomplete data for
> > > multiple transactions, then we can decide to spill the transaction or
> > > maybe we can directly spill the first largest transaction which has
> > > incomplete data.
> > >
> > Yeah, we might do something on this line. Basically, we need to mark
> > the top-transaction as data-incomplete if any of its subtransaction is
> > having data-incomplete (it will always be the latest sub-transaction
> > of the top transaction). Also, for streaming, we are checking the
> > largest top transaction whereas for spilling we just need the larget
> > (sub) transaction. So we also need to decide while picking the
> > largest top transaction for streaming, if we get a few transactions
> > with in-complete data then how we will go for the spill. Do we spill
> > all the sub-transactions under this top transaction or we will again
> > find the larget (sub) transaction for spilling.
> >
>
> I think it is better to do later as that will lead to the spill of
> only required (minimum changes to get the memory below threshold)
> changes.
I think instead of doing this can't we just spill the changes which
are in toast_hash. Basically, at the end of the stream, we have some
toast tuple which we could not stream because we did not have the
insert for the main table then we can spill only those changes which
are in tuple hash. And, in the subsequent stream whenever we get the
insert for the main table at that time we can restore those changes
and stream. We can also maintain some flag saying data is not
complete (with some change LSN number) and after that LSN we can spill
any toast change to disk until we get the change for the main table,
by this way we can avoid building tuple hash until we get the change
for the main table.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-14 05:13 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-14 05:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 13, 2020 at 3:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > > The problem is that when we
> > > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > > but don't stream until we get the actual change on the main table.
> > > > > Now, the problem is that we might get the change of the toasted table
> > > > > and the main table in different streams. So basically, in a stream,
> > > > > if we have only got the toasted tuples then even after
> > > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > > >
> > > >
> > > > I think we can't split such changes in a different stream (unless we
> > > > design an entirely new solution to send partial changes of toast
> > > > data), so we need to send them together. We can keep a flag like
> > > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > > are able to assemble the entire tuple. Now, whenever, we try to
> > > > stream the changes once we reach the memory threshold, we can check
> > > > whether the data_complete flag is true
Here, we can also consider streaming the changes when data_complete is
false, but some additional changes have been added to the same txn as
the new changes might complete the tuple.
> > > > , if so, then only send the
> > > > changes, otherwise, we can pick the next largest transaction. I think
> > > > we can retry it for few times and if we get the incomplete data for
> > > > multiple transactions, then we can decide to spill the transaction or
> > > > maybe we can directly spill the first largest transaction which has
> > > > incomplete data.
> > > >
> > > Yeah, we might do something on this line. Basically, we need to mark
> > > the top-transaction as data-incomplete if any of its subtransaction is
> > > having data-incomplete (it will always be the latest sub-transaction
> > > of the top transaction). Also, for streaming, we are checking the
> > > largest top transaction whereas for spilling we just need the larget
> > > (sub) transaction. So we also need to decide while picking the
> > > largest top transaction for streaming, if we get a few transactions
> > > with in-complete data then how we will go for the spill. Do we spill
> > > all the sub-transactions under this top transaction or we will again
> > > find the larget (sub) transaction for spilling.
> > >
> >
> > I think it is better to do later as that will lead to the spill of
> > only required (minimum changes to get the memory below threshold)
> > changes.
> I think instead of doing this can't we just spill the changes which
> are in toast_hash. Basically, at the end of the stream, we have some
> toast tuple which we could not stream because we did not have the
> insert for the main table then we can spill only those changes which
> are in tuple hash.
>
Hmm, I think this can turn out to be inefficient because we can easily
end up spilling the data even when we don't need to so. Consider
cases, where part of the streamed changes are for toast, and remaining
are the changes which we would have streamed and hence can be removed.
In such cases, we could have easily consumed remaining changes for
toast without spilling. Also, I am not sure if spilling changes from
the hash table is a good idea as they are no more in the same order as
they were in ReorderBuffer which means the order in which we serialize
the changes normally would change and that might have some impact, so
we would need some more study if we want to pursue this idea.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-21 16:36 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-01-21 16:36 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 14, 2020 at 10:56:37AM +0530, Dilip Kumar wrote:
>On Sat, Jan 11, 2020 at 3:07 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>>
>> On 2020-Jan-10, Alvaro Herrera wrote:
>>
>> > Here's a rebase of this patch series. I didn't change anything except
>>
>> ... this time with attachments ...
>The patch set fails to apply on the head so rebased. (Rebased on
>commit cebf9d6e6ee13cbf9f1a91ec633cf96780ffc985)
>
I've noticed the patch was in WoA state since 2019/12/01, but there's
been quite a lot of traffic on this thread and a bunch of new patch
versions. So I've switched it to "needs review" - if that's not the
right status, let me know.
Also, the patch was moved forward mostly by Amit and Dilip, so I've
added them as authors in the CF app (well, what matters is the commit
message, of course, but let's keep this up to date too).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-22 05:10 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-22 05:10 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
Update on the open items
> As per my understanding apart from the above comments, the known
> pending work for this patchset is as follows:
> a. The two open items agreed to you in the email [3]. -> The first part is done and the second part is an improvement, not a bugfix. I will try to work on this part in the next patch set.
> b. Complete the handling of schema_sent as discussed above [4]. -> Done
> c. Few comments by Vignesh and the response on the same by me [5][6]. -> Done
> d. WAL overhead and performance testing for additional WAL logging by
> this patchset. -> Pending
> e. Some way to see the tuple for streamed transactions by decoding API
> as speculated by you [7]. ->Pending
f. Bug in the toast table handling -> Submitted as a separate POC
patch, which can be merged to the main after review and more testing.
> [3] - https://www.postgresql.org/message-id/CAFiTN-uT5YZE0egGhKdTteTjcGrPi8hb%3DFMPpr9_hEB7hozQ-Q%40mail.g...
> [4] - https://www.postgresql.org/message-id/CAA4eK1KjD9x0mS4JxzCbu3gu-r6K7XJRV%2BZcGb3BH6U3x2uxew%40mail.g...
> [5] - https://www.postgresql.org/message-id/CALDaNm0DNUojjt7CV-fa59_kFbQQ3rcMBtauvo44ttea7r9KaA%40mail.gma...
> [6] - https://www.postgresql.org/message-id/CAA4eK1%2BZvupW00c--dqEg8f3dHZDOGmA9xOQLyQHjRSoDi6AkQ%40mail.g...
> [7] - https://www.postgresql.org/message-id/CAFiTN-t8PmKA1X4jEqKmkvs0ggWpy0APWpPuaJwpx2YpfAf97w%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-22 16:37 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-22 16:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
I looked at this patchset and it seemed natural to apply 0008 next
(adding work_mem to subscriptions). Attached is Dilip's latest version,
plus my review changes. This will break the patch tester's logic; sorry
about that.
What part of this change is what sets the process's
logical_decoding_work_mem to the given value? I was unable to figure
that out. Is it missing or am I just stupid?
Changes:
* the patch adds logical_decoding_work_mem SGML, but that has already
been applied (cec2edfa7859); remove dupe.
* parse_subscription_options() comment says that it will raise an error if a
caller does not pass the pointer for an option but option list
specifies that option. It does not really implement that behavior (an
existing problem): instead, if the pointer is not passed, the option
is ignored. Moreover, this new patch continued to fail to handle
things as the comment says. I decided to implement the documented
behavior instead; it's now inconsistent with how the other options are
implemented. I think we should fix the other options to behave as the
comment says, because it's a more convenient API; if we instead opted
to update the code comment to match the code, each caller would have
to be checked to verify that the correct options are passed, which is
pointless and error prone.
* the parse_subscription_options API is a mess. I reordered the
arguments a little bit; also change the argument layout in callers so
that each caller is grouped more sensibly. Also added comments to
simplify reading the argument lists. I think this could be fixed by
using an ad-hoc struct to pass in and out. Didn't get around to doing
that, seems an unrelated potential improvement.
* trying to do own range checking in pgoutput and subscriptioncmds.c
seems pointless and likely to get out of sync with guc.c. Simpler is
to call set_config_option() to verify that the argument is in range.
(Note a further problem in the patch series: the range check in
subscriptioncmds.c is only added in patch 0009).
* parsing integers using scanint8() seemed weird (error messages there
do not correspond to what we want). After a couple of false starts, I
decided to rely on guc.c's set_config_option() followed by parse_int().
That also has the benefit that you can give it units.
* psql \dRs+ should display the work_mem; patch failed to do that.
Added. Unit display is done by pg_size_pretty(), which might be
different from what guc.c does, but I think it works OK.
It's the first place where we use pg_size_pretty to show a memory
limit, however.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Dilip-s-original.patch (text/x-diff, 13.0K) download
From a31b4ebd90dd7a4c94a35f2b3452258078c30e37 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 22 Jan 2020 12:44:13 -0300
Subject: [PATCH 1/2] Dilip's original
---
doc/src/sgml/config.sgml | 21 +++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 ++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 3ccacd528b..163cc77d1d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..91790b0c95 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83bd2e..5cd1daa238 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9bfe142ada..c50e854e96 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +689,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -696,6 +720,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +738,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +777,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +814,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..896ddab2b1 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f95c..48b960c4c9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 752508213a..536722b32f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..3394379f86 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e08afc6548..4c7acfb7d3 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -169,6 +169,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1
0002-Changes-by-lvaro.patch (text/x-diff, 9.9K) download
From 848ad7383cede7600ae3fca07440e3f2441ac934 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 22 Jan 2020 12:51:28 -0300
Subject: [PATCH 2/2] =?UTF-8?q?Changes=20by=20=C3=81lvaro?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
doc/src/sgml/config.sgml | 21 -------
src/backend/commands/subscriptioncmds.c | 62 +++++++++++++--------
src/backend/replication/pgoutput/pgoutput.c | 23 +++-----
src/bin/psql/describe.c | 4 +-
src/include/catalog/pg_subscription.h | 1 -
5 files changed, 52 insertions(+), 59 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 163cc77d1d..3ccacd528b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,27 +1751,6 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
- <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
- <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
- <indexterm>
- <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
- </indexterm>
- </term>
- <listitem>
- <para>
- Specifies the maximum amount of memory to be used by logical decoding,
- before some of the decoded changes are either written to local disk.
- This limits the amount of memory used by logical streaming replication
- connections. It defaults to 64 megabytes (<literal>64MB</literal>).
- Since each replication connection only uses a single buffer of this size,
- and an installation normally doesn't have many such connections
- concurrently (as limited by <varname>max_wal_senders</varname>), it's
- safe to set this value significantly higher than <varname>work_mem</varname>,
- reducing the amount of decoded changes written to disk.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c50e854e96..7920e75bfa 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -54,12 +54,13 @@ static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
* accommodate that.
*/
static void
-parse_subscription_options(List *options, bool *connect, bool *enabled_given,
- bool *enabled, bool *create_slot,
+parse_subscription_options(List *options, bool *connect,
+ bool *enabled_given, bool *enabled,
+ bool *create_slot,
bool *slot_name_given, char **slot_name,
+ bool *logical_wm_given, int *logical_wm,
bool *copy_data, char **synchronous_commit,
- bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *refresh)
{
ListCell *lc;
bool connect_given = false;
@@ -177,15 +178,25 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
- else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ else if (strcmp(defel->defname, "work_mem") == 0)
{
+ if (!logical_wm)
+ elog(ERROR, "option \"work_mem\" not valid in this context");
+
if (*logical_wm_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("conflicting or redundant options")));
+ /* Test if the value is valid for logical_decoding_work_mem */
+ (void) set_config_option("logical_decoding_work_mem", defGetString(defel),
+ PGC_BACKEND, PGC_S_TEST, GUC_ACTION_SET,
+ false, 0, false);
+ if (!parse_int(defGetString(defel), logical_wm,
+ GUC_UNIT_KB, NULL))
+ elog(ERROR, "parse_int failed"); /* shouldn't happen */
*logical_wm_given = true;
- *logical_wm = defGetInt32(defel);
+
}
else
ereport(ERROR,
@@ -345,10 +356,11 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
*
* Connection and publication should not be specified here.
*/
- parse_subscription_options(stmt->options, &connect, &enabled_given,
- &enabled, &create_slot, &slotname_given,
- &slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ parse_subscription_options(stmt->options, &connect,
+ &enabled_given, &enabled,
+ &create_slot, &slotname_given, &slotname,
+ &logical_wm_given, &logical_wm,
+ ©_data, &synchronous_commit, NULL);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -692,10 +704,11 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
int logical_wm;
bool logical_wm_given;
- parse_subscription_options(stmt->options, NULL, NULL, NULL,
+ parse_subscription_options(stmt->options, NULL,
+ NULL, NULL, /* enabled */
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm_given, &logical_wm,
+ NULL, &synchronous_commit, NULL);
if (slotname_given)
{
@@ -737,9 +750,10 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
enabled_given;
parse_subscription_options(stmt->options, NULL,
- &enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ &enabled_given, &enabled,
+ NULL, NULL, NULL, /* slot */
+ NULL, NULL, /* logical wm */
+ NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -775,9 +789,11 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
bool copy_data;
bool refresh;
- parse_subscription_options(stmt->options, NULL, NULL, NULL,
- NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ parse_subscription_options(stmt->options, NULL,
+ NULL, NULL, /* enabled */
+ NULL, NULL, NULL, /* slot */
+ NULL, NULL, /* logical wm */
+ ©_data, NULL, &refresh);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -812,9 +828,11 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled subscriptions")));
- parse_subscription_options(stmt->options, NULL, NULL, NULL,
- NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ parse_subscription_options(stmt->options, NULL,
+ NULL, NULL, /* enabled */
+ NULL, NULL, NULL, /* slot */
+ NULL, NULL, /* logical wm */
+ ©_data, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b32f..d243d90821 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -13,6 +13,7 @@
#include "postgres.h"
#include "catalog/pg_publication.h"
+#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
@@ -141,26 +142,20 @@ parse_output_parameters(List *options, uint32 *protocol_version,
}
else if (strcmp(defel->defname, "work_mem") == 0)
{
- int64 parsed;
-
if (work_mem_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("conflicting or redundant options")));
work_mem_given = true;
+ /* Test if the value is valid for logical_decoding_work_mem */
+ (void) set_config_option("logical_decoding_work_mem", defGetString(defel),
+ PGC_BACKEND, PGC_S_TEST, GUC_ACTION_SET,
+ false, 0, false);
- if (!scanint8(strVal(defel->arg), true, &parsed))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid work_mem")));
-
- if (parsed > PG_INT32_MAX || parsed < 64)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("work_mem \"%s\" out of range",
- strVal(defel->arg))));
-
- *logical_decoding_work_mem = (int)parsed;
+ /* by here it must be valid, so this shouldn't fail */
+ if (!parse_int(defGetString(defel), logical_decoding_work_mem,
+ GUC_UNIT_KB, NULL))
+ elog(ERROR, "parse_int failed"); /* shouldn't happen */
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f3c7eb96fa..956ad41f56 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5933,7 +5933,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5961,8 +5961,10 @@ describeSubscriptions(const char *pattern, bool verbose)
{
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
+ ", pg_catalog.pg_size_pretty(subworkmem::bigint * 1024) AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
gettext_noop("Synchronous commit"),
+ gettext_noop("Working Memory"),
gettext_noop("Conninfo"));
}
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379f86..eef585b0e5 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -47,7 +47,6 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
-
int32 subworkmem; /* Memory to use to decode changes. */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
--
2.20.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-23 03:37 Amit Kapila <amit.kapila16@gmail.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-23 03:37 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jan 22, 2020 at 10:07 PM Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
>
> I looked at this patchset and it seemed natural to apply 0008 next
> (adding work_mem to subscriptions).
>
I am not so sure whether we need this patch as the exact scenario
where it can help is not very clear to me and neither did anyone
explained. I have raised this concern earlier as well [1]. The point
is that 'logical_decoding_work_mem' applies to the entire
ReorderBuffer in the publisher's side and how will a parameter from a
particular subscription help in that?
[1] - https://www.postgresql.org/message-id/CAA4eK1J%2B3kab6RSZrgj0YiQV1r%2BH3FWVaNjKhWvpEe5-bpZiBw%40mail...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 05:57 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 05:57 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Hmm, I think this can turn out to be inefficient because we can easily
> > end up spilling the data even when we don't need to so. Consider
> > cases, where part of the streamed changes are for toast, and remaining
> > are the changes which we would have streamed and hence can be removed.
> > In such cases, we could have easily consumed remaining changes for
> > toast without spilling. Also, I am not sure if spilling changes from
> > the hash table is a good idea as they are no more in the same order as
> > they were in ReorderBuffer which means the order in which we serialize
> > the changes normally would change and that might have some impact, so
> > we would need some more study if we want to pursue this idea.
> I have fixed this bug and attached it as a separate patch. I will
> merge it to the main patch after we agree with the idea and after some
> more testing.
>
> The idea is that whenever we get the toasted chunk instead of directly
> inserting it into the toast hash I am inserting it into some local
> list so that if we don't get the change for the main table then we can
> insert these changes back to the txn->changes. So once we get the
> change for the main table at that time I am preparing the hash table
> to merge the chunks.
>
I think this idea will work but appears to be quite costly because (a)
you might need to serialize/deserialize the changes multiple times and
might attempt streaming multiple times even though you can't do (b)
you need to remove/add the same set of changes from the main list
multiple times.
It seems to me that we need to add all of this new handling because
while taking the decision whether to stream or not we don't know
whether the txn has changes that can't be streamed. One idea to make
it work is that we identify it while decoding the WAL. I think we
need to set a bit in the insert/delete WAL record to identify if the
tuple belongs to a toast relation. This won't add any additional
overhead in WAL and reduce a lot of complexity in the logical decoding
and also decoding will be efficient. If this is feasible, then we can
do the same for speculative insertions.
In patch v8-0013-Bugfix-handling-of-incomplete-toast-tuple, why is
below change required?
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,3 +1,4 @@
wal_level = logical
max_replication_slots = 4
logical_decoding_work_mem = 64kB
+logging_collector=on
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 06:04 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 06:04 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Hmm, I think this can turn out to be inefficient because we can easily
> > > end up spilling the data even when we don't need to so. Consider
> > > cases, where part of the streamed changes are for toast, and remaining
> > > are the changes which we would have streamed and hence can be removed.
> > > In such cases, we could have easily consumed remaining changes for
> > > toast without spilling. Also, I am not sure if spilling changes from
> > > the hash table is a good idea as they are no more in the same order as
> > > they were in ReorderBuffer which means the order in which we serialize
> > > the changes normally would change and that might have some impact, so
> > > we would need some more study if we want to pursue this idea.
> > I have fixed this bug and attached it as a separate patch. I will
> > merge it to the main patch after we agree with the idea and after some
> > more testing.
> >
> > The idea is that whenever we get the toasted chunk instead of directly
> > inserting it into the toast hash I am inserting it into some local
> > list so that if we don't get the change for the main table then we can
> > insert these changes back to the txn->changes. So once we get the
> > change for the main table at that time I am preparing the hash table
> > to merge the chunks.
> >
>
>
> I think this idea will work but appears to be quite costly because (a)
> you might need to serialize/deserialize the changes multiple times and
> might attempt streaming multiple times even though you can't do (b)
> you need to remove/add the same set of changes from the main list
> multiple times.
I agree with this.
>
> It seems to me that we need to add all of this new handling because
> while taking the decision whether to stream or not we don't know
> whether the txn has changes that can't be streamed. One idea to make
> it work is that we identify it while decoding the WAL. I think we
> need to set a bit in the insert/delete WAL record to identify if the
> tuple belongs to a toast relation. This won't add any additional
> overhead in WAL and reduce a lot of complexity in the logical decoding
> and also decoding will be efficient. If this is feasible, then we can
> do the same for speculative insertions.
The Idea looks good to me. I will work on this.
>
> In patch v8-0013-Bugfix-handling-of-incomplete-toast-tuple, why is
> below change required?
>
> --- a/contrib/test_decoding/logical.conf
> +++ b/contrib/test_decoding/logical.conf
> @@ -1,3 +1,4 @@
> wal_level = logical
> max_replication_slots = 4
> logical_decoding_work_mem = 64kB
> +logging_collector=on
Sorry, these are some local changes which got included in the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 06:12 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 06:12 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > end up spilling the data even when we don't need to so. Consider
> > > > cases, where part of the streamed changes are for toast, and remaining
> > > > are the changes which we would have streamed and hence can be removed.
> > > > In such cases, we could have easily consumed remaining changes for
> > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > the hash table is a good idea as they are no more in the same order as
> > > > they were in ReorderBuffer which means the order in which we serialize
> > > > the changes normally would change and that might have some impact, so
> > > > we would need some more study if we want to pursue this idea.
> > > I have fixed this bug and attached it as a separate patch. I will
> > > merge it to the main patch after we agree with the idea and after some
> > > more testing.
> > >
> > > The idea is that whenever we get the toasted chunk instead of directly
> > > inserting it into the toast hash I am inserting it into some local
> > > list so that if we don't get the change for the main table then we can
> > > insert these changes back to the txn->changes. So once we get the
> > > change for the main table at that time I am preparing the hash table
> > > to merge the chunks.
> > >
> >
> >
> > I think this idea will work but appears to be quite costly because (a)
> > you might need to serialize/deserialize the changes multiple times and
> > might attempt streaming multiple times even though you can't do (b)
> > you need to remove/add the same set of changes from the main list
> > multiple times.
> I agree with this.
> >
> > It seems to me that we need to add all of this new handling because
> > while taking the decision whether to stream or not we don't know
> > whether the txn has changes that can't be streamed. One idea to make
> > it work is that we identify it while decoding the WAL. I think we
> > need to set a bit in the insert/delete WAL record to identify if the
> > tuple belongs to a toast relation. This won't add any additional
> > overhead in WAL and reduce a lot of complexity in the logical decoding
> > and also decoding will be efficient. If this is feasible, then we can
> > do the same for speculative insertions.
> The Idea looks good to me. I will work on this.
>
One more thing we can do is to identify whether the tuple belongs to
toast relation while decoding it. However, I think to do that we need
to have access to relcache at that time and that might add some
overhead as we need to do that for each tuple. Can we investigate
what it will take to do that and if it is better than setting a bit
during WAL logging.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 06:28 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 06:28 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > > end up spilling the data even when we don't need to so. Consider
> > > > > cases, where part of the streamed changes are for toast, and remaining
> > > > > are the changes which we would have streamed and hence can be removed.
> > > > > In such cases, we could have easily consumed remaining changes for
> > > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > > the hash table is a good idea as they are no more in the same order as
> > > > > they were in ReorderBuffer which means the order in which we serialize
> > > > > the changes normally would change and that might have some impact, so
> > > > > we would need some more study if we want to pursue this idea.
> > > > I have fixed this bug and attached it as a separate patch. I will
> > > > merge it to the main patch after we agree with the idea and after some
> > > > more testing.
> > > >
> > > > The idea is that whenever we get the toasted chunk instead of directly
> > > > inserting it into the toast hash I am inserting it into some local
> > > > list so that if we don't get the change for the main table then we can
> > > > insert these changes back to the txn->changes. So once we get the
> > > > change for the main table at that time I am preparing the hash table
> > > > to merge the chunks.
> > > >
> > >
> > >
> > > I think this idea will work but appears to be quite costly because (a)
> > > you might need to serialize/deserialize the changes multiple times and
> > > might attempt streaming multiple times even though you can't do (b)
> > > you need to remove/add the same set of changes from the main list
> > > multiple times.
> > I agree with this.
> > >
> > > It seems to me that we need to add all of this new handling because
> > > while taking the decision whether to stream or not we don't know
> > > whether the txn has changes that can't be streamed. One idea to make
> > > it work is that we identify it while decoding the WAL. I think we
> > > need to set a bit in the insert/delete WAL record to identify if the
> > > tuple belongs to a toast relation. This won't add any additional
> > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > and also decoding will be efficient. If this is feasible, then we can
> > > do the same for speculative insertions.
> > The Idea looks good to me. I will work on this.
> >
>
> One more thing we can do is to identify whether the tuple belongs to
> toast relation while decoding it. However, I think to do that we need
> to have access to relcache at that time and that might add some
> overhead as we need to do that for each tuple. Can we investigate
> what it will take to do that and if it is better than setting a bit
> during WAL logging.
IMHO, for the catalog scan, we will have to start/stop the transaction
for each change. So do you want that we should evaluate its
performance? Also, during we get the change we might not have the
complete historic snapshot ready to fetch the rel cache entry.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 08:00 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 08:00 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > > It seems to me that we need to add all of this new handling because
> > > > while taking the decision whether to stream or not we don't know
> > > > whether the txn has changes that can't be streamed. One idea to make
> > > > it work is that we identify it while decoding the WAL. I think we
> > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > tuple belongs to a toast relation. This won't add any additional
> > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > and also decoding will be efficient. If this is feasible, then we can
> > > > do the same for speculative insertions.
> > > The Idea looks good to me. I will work on this.
> > >
> >
> > One more thing we can do is to identify whether the tuple belongs to
> > toast relation while decoding it. However, I think to do that we need
> > to have access to relcache at that time and that might add some
> > overhead as we need to do that for each tuple. Can we investigate
> > what it will take to do that and if it is better than setting a bit
> > during WAL logging.
>
> IMHO, for the catalog scan, we will have to start/stop the transaction
> for each change. So do you want that we should evaluate its
> performance?
>
No, I was not thinking about each change, but at the level of ReorderBufferTXN.
> Also, during we get the change we might not have the
> complete historic snapshot ready to fetch the rel cache entry.
>
Before decoding each change (say DecodeInsert), we call
SnapBuildProcessChange. Isn't that sufficient?
Even, if the above is possible, I am not sure how good is it for each
change we fetch rel cache entry, that is the point I was worried.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 08:25 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 08:25 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 1:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > > > It seems to me that we need to add all of this new handling because
> > > > > while taking the decision whether to stream or not we don't know
> > > > > whether the txn has changes that can't be streamed. One idea to make
> > > > > it work is that we identify it while decoding the WAL. I think we
> > > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > > tuple belongs to a toast relation. This won't add any additional
> > > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > > and also decoding will be efficient. If this is feasible, then we can
> > > > > do the same for speculative insertions.
> > > > The Idea looks good to me. I will work on this.
> > > >
> > >
> > > One more thing we can do is to identify whether the tuple belongs to
> > > toast relation while decoding it. However, I think to do that we need
> > > to have access to relcache at that time and that might add some
> > > overhead as we need to do that for each tuple. Can we investigate
> > > what it will take to do that and if it is better than setting a bit
> > > during WAL logging.
> >
> > IMHO, for the catalog scan, we will have to start/stop the transaction
> > for each change. So do you want that we should evaluate its
> > performance?
> >
>
> No, I was not thinking about each change, but at the level of ReorderBufferTXN.
That means we will have to keep that transaction open until we decode
the commit WAL for that ReorderBufferTXN or you have anything else in
mind?
>
> > Also, during we get the change we might not have the
> > complete historic snapshot ready to fetch the rel cache entry.
> >
>
> Before decoding each change (say DecodeInsert), we call
> SnapBuildProcessChange. Isn't that sufficient?
Yeah, Right, we can get some recache entry based on the base snapshot.
And, that might be sufficient to know whether it's a toast relation or
not.
>
> Even, if the above is possible, I am not sure how good is it for each
> change we fetch rel cache entry, that is the point I was worried.
We might not need to scan the catalog every time, we might get it from
the cache itself.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-28 09:06 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 09:06 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 1:55 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 1:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > > > It seems to me that we need to add all of this new handling because
> > > > > > while taking the decision whether to stream or not we don't know
> > > > > > whether the txn has changes that can't be streamed. One idea to make
> > > > > > it work is that we identify it while decoding the WAL. I think we
> > > > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > > > tuple belongs to a toast relation. This won't add any additional
> > > > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > > > and also decoding will be efficient. If this is feasible, then we can
> > > > > > do the same for speculative insertions.
> > > > > The Idea looks good to me. I will work on this.
> > > > >
> > > >
> > > > One more thing we can do is to identify whether the tuple belongs to
> > > > toast relation while decoding it. However, I think to do that we need
> > > > to have access to relcache at that time and that might add some
> > > > overhead as we need to do that for each tuple. Can we investigate
> > > > what it will take to do that and if it is better than setting a bit
> > > > during WAL logging.
> > >
> > > IMHO, for the catalog scan, we will have to start/stop the transaction
> > > for each change. So do you want that we should evaluate its
> > > performance?
> > >
> >
> > No, I was not thinking about each change, but at the level of ReorderBufferTXN.
> That means we will have to keep that transaction open until we decode
> the commit WAL for that ReorderBufferTXN or you have anything else in
> mind?
>
or probably till we start streaming.
> >
> > > Also, during we get the change we might not have the
> > > complete historic snapshot ready to fetch the rel cache entry.
> > >
> >
> > Before decoding each change (say DecodeInsert), we call
> > SnapBuildProcessChange. Isn't that sufficient?
> Yeah, Right, we can get some recache entry based on the base snapshot.
> And, that might be sufficient to know whether it's a toast relation or
> not.
> >
> > Even, if the above is possible, I am not sure how good is it for each
> > change we fetch rel cache entry, that is the point I was worried.
>
> We might not need to scan the catalog every time, we might get it from
> the cache itself.
>
Right, but I am not completely sure if that is better than setting a
bit in WAL record for toast tuples.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-30 10:41 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-30 10:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Jan 10, 2020 at 10:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> >
> > Few more comments:
> > --------------------------------
> > v4-0007-Implement-streaming-mode-in-ReorderBuffer
> > 1.
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > + /*
> > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > + * information about
> > subtransactions, which could arrive after streaming start.
> > + */
> > + if (!txn->is_schema_sent)
> > + snapshot_now
> > = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > + txn,
> > command_id);
> > ..
> > }
> >
> > Why are we using base snapshot here instead of the snapshot we saved
> > the first time streaming has happened? And as mentioned in comments,
> > won't we need to consider the snapshots for subtransactions that
> > arrived after the last time we have streamed the changes?
> Fixed
>
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
"because after we there", you seem to forget a word between 'we' and
'there'. So as we are copying it now, does this mean it will consider
the snapshots for subtransactions that arrived after the last time we
have streamed the changes? If so, have you tested it and can we add
the same in comments.
Also, if we need to copy the snapshot here, then do we need to again
copy it in ReorderBufferProcessTXN(in below code and in catch block in
the same function).
{
..
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
..
}
> >
> > 4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
> > fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
> > especially to cover the case when it gets called due to memory
> > overflow (aka via ReorderBufferCheckMemoryLimit).
> We get origin_lsn during commit time so I am not sure how can we do
> that. I have also noticed that currently, we are not using origin_lsn
> on the subscriber side. I think need more investigation that if we
> want this then do we need to log it early.
>
Have you done any investigation of this point? You might want to look
at pg_replication_origin* APIs. Today, again looking at this code, I
think with current coding, it won't be used even when we encounter
commit record. Because ReorderBufferCommit calls
ReorderBufferStreamCommit which will make sure that origin_id and
origin_lsn is never sent. I think at least that should be fixed, if
not, probably, we need a comment with reasoning why we think it is
okay not to do in this case.
+ /*
+ * If we are streaming the in-progress transaction then Discard the
/Discard/discard
> >
> > v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
> > 1.
> > + /*
> > + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> > + * error out
> > + */
> > + if (TransactionIdIsValid(CheckXidAlive) &&
> > + !TransactionIdIsInProgress(CheckXidAlive) &&
> > + !TransactionIdDidCommit(CheckXidAlive))
> > + ereport(ERROR,
> > + (errcode(ERRCODE_TRANSACTION_ROLLBACK),
> > + errmsg("transaction aborted during system catalog scan")));
> >
> > Why here we can't use TransactionIdDidAbort? If we can't use it, then
> > can you add comments stating the reason of the same.
> Done
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
How about replacing the above comment with below one:
If CheckXidAlive is valid, then we check if it aborted. If it did, we
error out. We can't directly use TransactionIdDidAbort as after crash
such transaction might not have been marked as aborted. See detailed
comments at snapmgr.c where the variable is declared.
I am not able to understand the change in
v8-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup. Do you have
any explanation for the same?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-30 12:39 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-30 12:39 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 10, 2020 at 10:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> >
> > >
> > > Few more comments:
> > > --------------------------------
> > > v4-0007-Implement-streaming-mode-in-ReorderBuffer
> > > 1.
> > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > {
> > > ..
> > > + /*
> > > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > > + * information about
> > > subtransactions, which could arrive after streaming start.
> > > + */
> > > + if (!txn->is_schema_sent)
> > > + snapshot_now
> > > = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > > + txn,
> > > command_id);
> > > ..
> > > }
> > >
> > > Why are we using base snapshot here instead of the snapshot we saved
> > > the first time streaming has happened? And as mentioned in comments,
> > > won't we need to consider the snapshots for subtransactions that
> > > arrived after the last time we have streamed the changes?
> > Fixed
> >
>
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * We can not use txn->snapshot_now directly because after we there
> + * might be some new sub-transaction which after the last streaming run
> + * so we need to add those sub-xip in the snapshot.
> + */
> + snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
> + txn, command_id);
>
> "because after we there", you seem to forget a word between 'we' and
> 'there'. So as we are copying it now, does this mean it will consider
> the snapshots for subtransactions that arrived after the last time we
> have streamed the changes? If so, have you tested it and can we add
> the same in comments.
Ok
> Also, if we need to copy the snapshot here, then do we need to again
> copy it in ReorderBufferProcessTXN(in below code and in catch block in
> the same function).
I think so because as part of the
"REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
point to the snapshot and that will get truncated when we truncate all
the changes of the ReorderBufferTXN. So I think we can check if
snapshot_now->copied is true then we can avoid copying otherwise we
can copy?
Other comments look fine to me so I will reply to them along with the
next version of the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-01-31 02:38 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-31 02:38 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Also, if we need to copy the snapshot here, then do we need to again
> > copy it in ReorderBufferProcessTXN(in below code and in catch block in
> > the same function).
> I think so because as part of the
> "REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
> point to the snapshot and that will get truncated when we truncate all
> the changes of the ReorderBufferTXN. So I think we can check if
> snapshot_now->copied is true then we can avoid copying otherwise we
> can copy?
>
Yeah, that makes sense, but I think then we also need to ensure that
ReorderBufferStreamTXN frees the snapshot only when it is copied. It
seems to me it should be always copied in the place where we are
trying to free it, so probably we should have an Assert there.
One more thing:
ReorderBufferProcessTXN()
{
..
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
..
}
I think it is better to move the above code into an inline function
(something like SetXidAlive). It will make the code in function
ReorderBufferProcessTXN look cleaner and easier to understand.
> Other comments look fine to me so I will reply to them along with the
> next version of the patch.
>
Okay, thanks.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-01 05:35 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-02-01 05:35 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> Other comments look fine to me so I will reply to them along with the
> next version of the patch.
>
This still needs more work, so I have moved this to the next CF.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-03 04:21 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-02-03 04:21 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Jan 10, 2020 at 10:53 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Dec 30, 2019 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > > 2. During commit time in DecodeCommit we check whether we need to skip
> > > the changes of the transaction or not by calling
> > > SnapBuildXactNeedsSkip but since now we support streaming so it's
> > > possible that before we decode the commit WAL, we might have already
> > > sent the changes to the output plugin even though we could have
> > > skipped those changes. So my question is instead of checking at the
> > > commit time can't we check before adding to ReorderBuffer itself
> > >
> >
> > I think if we can do that then the same will be true for current code
> > irrespective of this patch. I think it is possible that we can't take
> > that decision while decoding because we haven't assembled a consistent
> > snapshot yet. I think we might be able to do that while we try to
> > stream the changes. I think we need to take care of all the
> > conditions during streaming (when the logical_decoding_workmem limit
> > is reached) as we do in DecodeCommit. This needs a bit more study.
>
> I have analyzed this further and I think we can not decide all the
> conditions even while streaming. Because IMHO once we get the
> SNAPBUILD_FULL_SNAPSHOT we can add the changes to the reorder buffer
> so that if we get the commit of the transaction after we reach to the
> SNAPBUILD_CONSISTENT. However, if we get the commit before we reach
> to SNAPBUILD_CONSISTENT then we need to ignore this transaction. Now,
> even if we have SNAPBUILD_FULL_SNAPSHOT we can stream the changes
> which might get dropped later but that we can not decide while
> streaming.
>
This makes sense to me, but we should add a comment for the same when
we are streaming to say we can't skip similar to how we do during
commit time because of the above reason described by you. Also, what
about other conditions where we can skip the transaction, basically
cases like (a) when the transaction happened in another database, (b)
when the output plugin is not interested in the origin and (c) when we
are doing fast-forwarding
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-04 05:20 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-02-04 05:20 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Feb 3, 2020 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 10, 2020 at 10:53 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Dec 30, 2019 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > > 2. During commit time in DecodeCommit we check whether we need to skip
> > > > the changes of the transaction or not by calling
> > > > SnapBuildXactNeedsSkip but since now we support streaming so it's
> > > > possible that before we decode the commit WAL, we might have already
> > > > sent the changes to the output plugin even though we could have
> > > > skipped those changes. So my question is instead of checking at the
> > > > commit time can't we check before adding to ReorderBuffer itself
> > > >
> > >
> > > I think if we can do that then the same will be true for current code
> > > irrespective of this patch. I think it is possible that we can't take
> > > that decision while decoding because we haven't assembled a consistent
> > > snapshot yet. I think we might be able to do that while we try to
> > > stream the changes. I think we need to take care of all the
> > > conditions during streaming (when the logical_decoding_workmem limit
> > > is reached) as we do in DecodeCommit. This needs a bit more study.
> >
> > I have analyzed this further and I think we can not decide all the
> > conditions even while streaming. Because IMHO once we get the
> > SNAPBUILD_FULL_SNAPSHOT we can add the changes to the reorder buffer
> > so that if we get the commit of the transaction after we reach to the
> > SNAPBUILD_CONSISTENT. However, if we get the commit before we reach
> > to SNAPBUILD_CONSISTENT then we need to ignore this transaction. Now,
> > even if we have SNAPBUILD_FULL_SNAPSHOT we can stream the changes
> > which might get dropped later but that we can not decide while
> > streaming.
> >
>
> This makes sense to me, but we should add a comment for the same when
> we are streaming to say we can't skip similar to how we do during
> commit time because of the above reason described by you. Also, what
> about other conditions where we can skip the transaction, basically
> cases like (a) when the transaction happened in another database, (b)
> when the output plugin is not interested in the origin and (c) when we
> are doing fast-forwarding
I will analyze those and fix in my next version of the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-04 05:29 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-04 05:29 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > > end up spilling the data even when we don't need to so. Consider
> > > > > cases, where part of the streamed changes are for toast, and remaining
> > > > > are the changes which we would have streamed and hence can be removed.
> > > > > In such cases, we could have easily consumed remaining changes for
> > > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > > the hash table is a good idea as they are no more in the same order as
> > > > > they were in ReorderBuffer which means the order in which we serialize
> > > > > the changes normally would change and that might have some impact, so
> > > > > we would need some more study if we want to pursue this idea.
> > > > I have fixed this bug and attached it as a separate patch. I will
> > > > merge it to the main patch after we agree with the idea and after some
> > > > more testing.
> > > >
> > > > The idea is that whenever we get the toasted chunk instead of directly
> > > > inserting it into the toast hash I am inserting it into some local
> > > > list so that if we don't get the change for the main table then we can
> > > > insert these changes back to the txn->changes. So once we get the
> > > > change for the main table at that time I am preparing the hash table
> > > > to merge the chunks.
> > > >
> > >
> > >
> > > I think this idea will work but appears to be quite costly because (a)
> > > you might need to serialize/deserialize the changes multiple times and
> > > might attempt streaming multiple times even though you can't do (b)
> > > you need to remove/add the same set of changes from the main list
> > > multiple times.
> > I agree with this.
> > >
> > > It seems to me that we need to add all of this new handling because
> > > while taking the decision whether to stream or not we don't know
> > > whether the txn has changes that can't be streamed. One idea to make
> > > it work is that we identify it while decoding the WAL. I think we
> > > need to set a bit in the insert/delete WAL record to identify if the
> > > tuple belongs to a toast relation. This won't add any additional
> > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > and also decoding will be efficient. If this is feasible, then we can
> > > do the same for speculative insertions.
> > The Idea looks good to me. I will work on this.
> >
>
> One more thing we can do is to identify whether the tuple belongs to
> toast relation while decoding it. However, I think to do that we need
> to have access to relcache at that time and that might add some
> overhead as we need to do that for each tuple. Can we investigate
> what it will take to do that and if it is better than setting a bit
> during WAL logging.
>
I have done some more analysis on this and it appears that there are
few problems in doing this. Basically, once we get the confirmed
flush location, we advance the replication_slot_catalog_xmin so that
vacuum can garbage collect the old tuple. So the problem is that
while we are collecting the changes in the ReorderBuffer our catalog
version might have removed, and we might not find any relation entry
with that relfilenodeid (because it is dropped or altered in the
future).
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-05 03:57 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-02-05 03:57 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Feb 4, 2020 at 11:00 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > One more thing we can do is to identify whether the tuple belongs to
> > toast relation while decoding it. However, I think to do that we need
> > to have access to relcache at that time and that might add some
> > overhead as we need to do that for each tuple. Can we investigate
> > what it will take to do that and if it is better than setting a bit
> > during WAL logging.
> >
> I have done some more analysis on this and it appears that there are
> few problems in doing this. Basically, once we get the confirmed
> flush location, we advance the replication_slot_catalog_xmin so that
> vacuum can garbage collect the old tuple. So the problem is that
> while we are collecting the changes in the ReorderBuffer our catalog
> version might have removed, and we might not find any relation entry
> with that relfilenodeid (because it is dropped or altered in the
> future).
>
Hmm, this means this can also occur while streaming the changes. The
main reason as I understand is that it is because before decoding
commit, we don't know whether these changes are already sent to the
subscriber (based on confirmed_flush_location/start_decoding_at). I
think it is better to skip streaming such transactions as we can't
make the right decision about these and as this can happen generally
after the crash for the first few transactions, it shouldn't matter
much if we serialize such transactions instead of streaming them.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-05 04:15 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-05 04:15 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Jan 31, 2020 at 8:08 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Also, if we need to copy the snapshot here, then do we need to again
> > > copy it in ReorderBufferProcessTXN(in below code and in catch block in
> > > the same function).
> > I think so because as part of the
> > "REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
> > point to the snapshot and that will get truncated when we truncate all
> > the changes of the ReorderBufferTXN. So I think we can check if
> > snapshot_now->copied is true then we can avoid copying otherwise we
> > can copy?
> >
>
> Yeah, that makes sense, but I think then we also need to ensure that
> ReorderBufferStreamTXN frees the snapshot only when it is copied. It
> seems to me it should be always copied in the place where we are
> trying to free it, so probably we should have an Assert there.
>
> One more thing:
> ReorderBufferProcessTXN()
> {
> ..
> + if (streaming)
> + {
> + /*
> + * While streaming an in-progress transaction there is a
> + * possibility that the (sub)transaction might get aborted
> + * concurrently. In such case if the (sub)transaction has
> + * catalog update then we might decode the tuple using wrong
> + * catalog version. So for detecting the concurrent abort we
> + * set CheckXidAlive to the current (sub)transaction's xid for
> + * which this change belongs to. And, during catalog scan we
> + * can check the status of the xid and if it is aborted we will
> + * report an specific error which we can ignore. We might have
> + * already streamed some of the changes for the aborted
> + * (sub)transaction, but that is fine because when we decode the
> + * abort we will stream abort message to truncate the changes in
> + * the subscriber.
> + */
> + CheckXidAlive = change->txn->xid;
> + }
> ..
> }
>
> I think it is better to move the above code into an inline function
> (something like SetXidAlive). It will make the code in function
> ReorderBufferProcessTXN look cleaner and easier to understand.
>
Fixed in the latest version sent upthread.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-05 04:18 Dilip Kumar <dilipbalaut@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-02-05 04:18 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 9:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Feb 4, 2020 at 11:00 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > One more thing we can do is to identify whether the tuple belongs to
> > > toast relation while decoding it. However, I think to do that we need
> > > to have access to relcache at that time and that might add some
> > > overhead as we need to do that for each tuple. Can we investigate
> > > what it will take to do that and if it is better than setting a bit
> > > during WAL logging.
> > >
> > I have done some more analysis on this and it appears that there are
> > few problems in doing this. Basically, once we get the confirmed
> > flush location, we advance the replication_slot_catalog_xmin so that
> > vacuum can garbage collect the old tuple. So the problem is that
> > while we are collecting the changes in the ReorderBuffer our catalog
> > version might have removed, and we might not find any relation entry
> > with that relfilenodeid (because it is dropped or altered in the
> > future).
> >
>
> Hmm, this means this can also occur while streaming the changes. The
> main reason as I understand is that it is because before decoding
> commit, we don't know whether these changes are already sent to the
> subscriber (based on confirmed_flush_location/start_decoding_at).
Right.
>I think it is better to skip streaming such transactions as we can't
> make the right decision about these and as this can happen generally
> after the crash for the first few transactions, it shouldn't matter
> much if we serialize such transactions instead of streaming them.
I think the idea makes sense to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2020-02-06 09:21 Amit Kapila <amit.kapila16@gmail.com>
parent: Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-02-06 09:21 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 9:42 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> >
> > I am not able to understand the change in
> > v8-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup. Do you have
> > any explanation for the same?
>
> It appears that in ReorderBufferCommitChild we are always setting the
> final_lsn of the subxacts so it should not be invalid. For testing, I
> have changed this as an assert and checked but it never hit. So maybe
> we can remove this change.
>
Tomas, do you remember anything about this change? We are talking
about below change:
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v8 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c
b/src/backend/replication/logical/reorderbuffer.c
index fe4e57c..beb6cd2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb,
ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
283 further messages in this thread omitted from this page (render size limit).
Use the per-message pages, or the t.mbox.gz byte stream, for the complete thread.
end of thread, other threads:[~2020-08-28 10:16 UTC | newest]
Thread overview: 536+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Erik Rijkers <er@xs4all.nl>
2017-12-25 17:40 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-25 22:08 ` Erik Rijkers <er@xs4all.nl>
2017-12-26 14:50 ` Dmitry Dolgov <9erthalion6@gmail.com>
2017-12-24 04:51 ` Craig Ringer <craig@2ndquadrant.com>
2017-12-24 13:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 17:30 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-29 17:56 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 13:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-10 10:27 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-22 19:02 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-23 10:50 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-30 04:08 ` vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 10:05 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-12 10:42 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-11-12 11:34 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 11:51 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-05 05:38 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-06 11:03 ` vignesh C <vignesh21@gmail.com>
2019-11-13 12:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-13 10:05 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-30 09:40 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 07:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-13 09:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 08:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 08:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 09:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:29 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 03:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-11 02:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-14 05:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-21 16:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-01-30 10:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-07 10:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-10 08:22 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-10 09:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-11 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-03 21:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-05 17:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-28 06:25 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Erik Rijkers <er@xs4all.nl>
2020-04-15 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:33 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 14:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-21 12:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 05:45 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:07 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:20 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-27 10:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-27 11:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 03:18 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-29 09:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-30 07:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-01 15:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-04 11:46 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 03:57 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 04:54 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 05:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 05:31 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 10:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 13:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-07 12:46 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-12 11:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-13 06:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-13 11:20 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-13 15:46 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-14 05:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 09:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 12:31 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 06:24 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 11:16 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-25 14:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-25 14:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-25 15:18 ` Erik Rijkers <er@xs4all.nl>
2020-05-26 02:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-27 14:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 11:52 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-02 10:28 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-02 11:25 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-02 14:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-03 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-04 08:35 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-07 11:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 09:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 10:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-10 11:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 11:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-12 06:08 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-12 11:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-15 03:41 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-15 12:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-16 09:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-17 10:26 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 06:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 10:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 06:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-29 15:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-03 10:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-05 06:07 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-07 11:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-08 06:23 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-09 09:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-16 14:19 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-17 04:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-17 04:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-18 15:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-18 15:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-22 10:56 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 11:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-29 10:54 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-30 03:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-30 04:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-30 11:50 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-04 06:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-05 11:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-06 06:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-06 06:13 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-06 09:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-07 15:29 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-12 16:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 05:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 06:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-15 04:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 11:21 ` Ajin Cherian <itsajin@gmail.com>
2020-07-15 11:23 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 10:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 04:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 05:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 05:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 09:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 09:26 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 09:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 10:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 10:38 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-14 12:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-15 03:59 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 13:29 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-16 06:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-16 10:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-20 06:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-20 08:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-20 11:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-20 13:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-21 03:37 ` Ajin Cherian <itsajin@gmail.com>
2020-07-22 03:48 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-22 04:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-22 10:50 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-22 11:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-23 06:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-24 11:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-24 13:47 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-25 11:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-26 05:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-26 13:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-28 04:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-29 05:16 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-30 06:57 ` Ajin Cherian <itsajin@gmail.com>
2020-07-30 09:53 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-31 09:52 ` Ajin Cherian <itsajin@gmail.com>
2020-08-04 04:42 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-04 07:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-05 12:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-05 14:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-06 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-07 08:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-13 06:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-13 13:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-15 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-17 12:58 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-19 04:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-19 06:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-19 08:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-20 08:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-20 08:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-20 12:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 03:44 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 03:58 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 05:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 05:03 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 09:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 11:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-22 03:08 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-24 16:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-25 04:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-25 05:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-25 12:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-25 14:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-26 17:52 ` Jeff Janes <jeff.janes@gmail.com>
2020-08-27 05:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-28 08:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-28 10:16 ` Neha Sharma <neha.sharma@enterprisedb.com>
2020-08-19 08:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-14 04:40 ` Thomas Munro <thomas.munro@gmail.com>
2020-08-14 06:14 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-14 22:43 ` Thomas Munro <thomas.munro@gmail.com>
2020-08-15 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-15 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 04:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-05 11:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-05 15:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-08 04:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-08 10:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-08 14:01 ` Ajin Cherian <itsajin@gmail.com>
2020-07-09 02:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-09 02:48 ` Ajin Cherian <itsajin@gmail.com>
2020-07-09 03:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-09 03:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 03:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 03:51 ` Ajin Cherian <itsajin@gmail.com>
2020-07-10 05:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-10 05:31 ` Ajin Cherian <itsajin@gmail.com>
2020-07-10 06:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 09:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 11:15 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 11:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-22 11:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 13:08 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-23 02:48 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-23 04:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-23 13:30 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 10:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-24 10:57 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 11:08 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-25 13:40 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-26 05:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-26 05:51 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-26 06:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-28 15:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-28 15:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 09:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 07:27 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 09:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 10:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 09:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 10:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 10:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 11:05 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 11:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-17 07:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-18 10:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-18 12:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-19 09:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-19 10:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 11:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 04:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-26 09:13 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 11:16 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-27 11:49 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-05-29 10:22 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-04 11:35 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-06-09 10:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-09 10:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 06:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 07:16 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 09:11 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 10:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 12:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 09:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-14 10:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 10:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-14 10:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 21:25 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-14 06:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 04:58 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 09:03 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 09:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-05 17:50 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-06 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-01 05:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-06 09:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-22 05:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-22 16:37 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-23 03:37 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-10 04:22 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-10 04:53 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-28 16:03 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-12-29 08:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 10:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-02 05:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 05:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-03 04:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:20 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 04:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-04 06:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 20:56 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-12-11 18:16 ` Robert Haas <robertmhaas@gmail.com>
2019-12-12 11:41 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-24 05:28 ` Robert Haas <robertmhaas@gmail.com>
2020-01-03 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-13 09:16 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-20 08:29 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2019-12-20 13:42 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-20 06:17 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-20 13:30 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-24 05:46 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-24 08:21 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-26 07:05 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-30 10:19 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-22 11:34 ` vignesh C <vignesh21@gmail.com>
2019-12-24 04:24 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-03 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-13 06:54 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-13 11:46 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-14 01:21 ` Craig Ringer <craig@2ndquadrant.com>
2019-10-14 02:11 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-14 09:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-18 12:01 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-21 05:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-21 09:20 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-21 09:41 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:12 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-23 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-24 13:37 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-04 10:16 ` vignesh C <vignesh21@gmail.com>
2019-11-06 06:03 ` vignesh C <vignesh21@gmail.com>
2019-11-07 09:49 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-07 10:19 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-07 11:43 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-16 13:14 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-19 11:55 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-21 10:44 ` Amit Kapila <amit.kapila16@gmail.com>
2018-01-03 20:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-03 20:13 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:36 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-19 14:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-19 22:08 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-31 06:53 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-02-01 22:50 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 00:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 05:19 ` Erik Rijkers <er@xs4all.nl>
2018-03-03 14:52 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-09 16:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-07-01 11:43 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-12-16 14:31 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-16 17:54 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-17 16:23 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2018-12-17 22:28 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-18 14:07 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2018-12-18 23:56 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-19 09:58 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-01-14 18:23 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-04 05:51 ` Michael Paquier <michael@paquier.xyz>
2019-02-04 13:49 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-28 17:17 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-28 19:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-29 14:37 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-29 18:48 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-30 15:59 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-16 16:54 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-16 19:29 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-16 20:57 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-10-02 04:59 ` Michael Paquier <michael@paquier.xyz>
2018-01-11 19:41 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-11 23:23 ` Greg Stark <stark@mit.edu>
2018-01-12 16:35 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-13 04:19 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-18 13:24 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-02-01 14:51 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-02-01 22:51 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-02 01:12 ` Andres Freund <andres@anarazel.de>
2018-03-02 02:33 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-02 02:39 ` David Steele <david@pgmasters.net>
2018-03-02 20:05 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:00 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 01:01 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 01:36 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:37 ` David Steele <david@pgmasters.net>
2018-03-03 01:54 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 02:08 ` David Steele <david@pgmasters.net>
2018-03-02 20:06 ` Robert Haas <robertmhaas@gmail.com>
2018-03-02 20:21 ` David Steele <david@pgmasters.net>
2018-03-02 21:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-13 04:01 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-29 16:34 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-02 22:06 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-03 10:39 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 13:28 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-26 18:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-27 05:48 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-25 13:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-26 18:07 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 19:33 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 19:36 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 21:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 21:46 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-27 09:40 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 12:03 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 04:34 ` Dilip Kumar <dilipbalaut@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox